{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How to Read Pytorch\n",
    "===================\n",
    "\n",
    "These five python notebooks are an introduction to core pytorch idioms.\n",
    "\n",
    "Pytorch is a numerical library that makes it very convenient to train deep networks on GPU hardware. It introduces a new programming vocabulary that takes a few steps beyond regular numerical python code. Although pytorch code can look simple and concrete, much of of the subtlety of what happens is invisible, so when working with pytorch code it helps to thoroughly understand the runtime model.\n",
    "\n",
    "For example, consider this code:\n",
    "\n",
    "```\n",
    "torch.nn.cross_entropy(model(images.cuda()), labels.cuda()).backward()\n",
    "optimizer.step()\n",
    "```\n",
    "\n",
    "It looks like it computes some function of `images` and `labels` without storing the answer.  But actually the purpose of this code is to update some hidden parameters that are not explicit in this formula.  This line of code moves batches of image and label data from CPU to the GPU; runs a neural network to make a prediction; constructs a computation graph describing how the loss depends on the network parameters; annotates every network parameter with a gradient; then finally it runs one step of optimization to adjust every parameter of the model.  During all this, the CPU does not see any of the actual answers.  That is intentional for speed reasons.  All the numerical computation is done on the GPU asynchronously and kept there.\n",
    "\n",
    "The berevity of the code is what makes pytorch code fun to write.  But it also reflects why pytorch can be so fast even though the python interpreter is so slow. Although the main python logic slogs along sequentially in a single very slow CPU thread, just a few python instructions can load a huge amount of work into the GPU.  That means the program can keep the GPU busy churning through massive numerical computations, for most part, without waiting for the python interpreter.\n",
    "\n",
    "Is is worth understanding five core idioms that work together to make this possible.  This tutorial has five Colab notebooks, one for each topic:\n",
    "\n",
    " 1. GPU Tensor arithmetic ([this notebook on colab](https://colab.research.google.com/github/davidbau/how-to-read-pytorch/blob/master/notebooks/1-Pytorch-Introduction.ipynb)): the notation for manipulating n-dimensional arrays of numbers on CPU or GPU.\n",
    " 2. [Autograd](./2-Pytorch-Autograd.ipynb): how to build a tensor computation graph and use it to get derivatives of any scalar with respect to any input.\n",
    " 3. [Optimization](./3-Pytorch-Optimizers.ipynb): ways to update tensor parameters to reduce any computed objective, using autograd gradients.\n",
    " 4. [Network modules](./4-Pytorch-Modules.ipynb): how pytorch represents neural networks for convenient composition, training, and saving.\n",
    " 5. [Datasets and Dataloaders](./5-Pytorch-Dataloader.ipynb): for efficient multithreaded prefetching of large streams of data.\n",
    "\n",
    "The key ideas are illustrated with small, illustrated, hackable examples, and there are links to other reference material and resources.\n",
    "\n",
    "All the notebooks can be run on Google Colab where some GPU compuation can be used for free, or they can be run on your own local Jupyter notebook server.\n",
    "\n",
    "The examples should all work with python 3.5 or newer and pytorch 1.0 or newer.\n",
    "\n",
    "The original [code on github can be found here](https://github.com/davidbau/how-to-read-pytorch).\n",
    "\n",
    "--- [*David Bau, July 2020*](http://davidbau.com/archives/2020/07/05/davids_tips_on_how_to_read_pytorch.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Topic 1: pytorch Tensors\n",
    "===============\n",
    "\n",
    "The first big trick for doing math fast on a modern computer is to do giant array operations all at once.\n",
    "\n",
    "To faciliate this, pytorch provides a [`torch.Tensor`](https://pytorch.org/docs/stable/tensors.html) class that is a lookalike to the older python numerical library [`numpy.ndarray`](https://numpy.org/doc/1.18/reference/arrays.ndarray.html).  Just like a numpy `ndarray`, the pytorch `Tensor` stores a d-dimensional array of numbers, where d can be zero or more, and where the contained numbers can be any of the usual selection of float or integer types.  Pytorch is designed to feel just like numpy: almost all the numpy operations are also available on torch tensors. But if something is missing, torch tensors can be directly converted to and from numpy using `x.numpy()` and `torch.from_numpy(a)`. So what is different and why did the pytorch authors bother to reimplement this whole library?\n",
    "\n",
    "**There are two things that pytorch Tensors have that numpy arrays lack:**\n",
    "\n",
    " 1. pytorch Tensors can live on either **GPU or CPU** (numpy is cpu-only).\n",
    " 2. pytorch can automatically track tensor computations to enable **automatic differentiation**.\n",
    "\n",
    "In the following sections on this page we talk about the basics of the Tensor API as well as point (1) - how to work with GPU and CPU tensors.  A discussion of (2) can be found in the next notebook, [2. Autograd](https://colab.research.google.com/github/davidbau/pytorch-tutorial/blob/master/notebooks/2-Pytorch-Autograd.ipynb).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Basic operations in the Tensor API\n",
    "----------------------------------\n",
    "\n",
    "Pytorch is not very different from numpy, although the pytorch API has more convenience methods such as `x.clamp(0).pow(2)` (supporting a chained method style, as is popular in Javascript libraries, so you don't need to say the verbose `numpy.clip(numpy.pow(x, 2), 0)`). So code is often shorter in pytorch.  A brief overview:\n",
    "\n",
    "**Elementwise operations.** Most tensor operations are simple (embarassingly parallelizable) elementwise operations, where the same math is done on every element of the array.  `x+y`, `x*y`, `x.abs()`, `x.pow(3)`, etc.  Unlike Matlab, `*` is for element-wise multiplication, not matrix-multiplication.\n",
    "\n",
    "**Copy semantics by default.** Almost all operations, including things like `x.sort()`, return a new copy of the tensor without overwriting the input tensors.  The exceptions are functions that end in an underscore such as `x.mul_(2)` which doubles the contents of x in-place.\n",
    "\n",
    "**Common reduction operations.**  There are some common operations such as `max`, `min`, `mean`, `sum` that reduce the array by one or more dimension. In pytorch, you can specify which dimension you want to reduce by passing the argument `dim=n`.\n",
    "\n",
    "**Why does min return two things?** Note that `[data, indexes] = x.sort(dim=0)` and `[vals, indexes] = x.min(dim=0)` return the pair of both the answer and the index values, so you do not need to separately recompute `argsort` or `argmin` when you need to know where the min came from.\n",
    "\n",
    "**What about linear algebra?** It's there.  `torch.mm(a,b)` is matrix multiplication, `torch.inverse(a)` inverts, `torch.eig(a)` gets eigenvalues, etc.\n",
    "\n",
    "The other thing to know is that pytorch tends to be very fast, often much faster than numpy even on CPU, because its implementation is aggressively parallelized behind-the-scenes.  Pytorch is willing to use multiple threads in situations where numpy just uses one.\n",
    "\n",
    "See the [reference for Tensor methods](https://pytorch.org/docs/stable/tensors.html#torch.Tensor) for what comes built-in.  A simple demo of some vectors:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([0.0000, 0.0500, 0.1000, 0.1500, 0.2000])\n"
     ]
    }
   ],
   "source": [
    "import math, numpy, torch\n",
    "from matplotlib import pyplot as plt\n",
    "\n",
    "# Make a vector of 101 equally spaced numbers from 0 to 5.\n",
    "x = torch.linspace(0, 5, 101)\n",
    "\n",
    "# Print the first five things in x.\n",
    "print(x[:5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise\n",
    "\n",
    "Print the last five things in x."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TODO\n"
     ]
    }
   ],
   "source": [
    "# TODO: Print the last five things in x (instead of the first five)\n",
    "\n",
    "print('TODO')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The shape of x is torch.Size([101])\n",
      "The shape of y1=x.sin() is torch.Size([101])\n",
      "The shape of y2=x ** x.cos() is torch.Size([101])\n",
      "The shape of y3=y2 - y1 is torch.Size([101])\n",
      "The shape of y4=y3.min() is torch.Size([]), a zero-d scalar\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAD4CAYAAADvsV2wAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAA5vElEQVR4nO3dd3hU1dbA4d9OgST0EpASCB0BKYIgRUARRaQIiBUEpXzY69ULKhd75YrtKtiQDiLSQRGkSQ1VuvQOAUIJgZSZ9f2xg7QAKTNzZpL1Ps95MjPnzDlrksyaPbsaEUEppVT2F+R0AEoppXxDE75SSuUQmvCVUiqH0ISvlFI5hCZ8pZTKIUKcDuBqihYtKtHR0U6HoZRSAWPFihVHRCQyrX1+nfCjo6OJiYlxOgyllAoYxphdV9qnVTpKKZVDaMJXSqkcQhO+UkrlEJrwlVIqh9CEr5RSOYQmfKWUyiE04SulVA6R5YRvjIkyxvxhjNlojFlvjHk2jWOaG2NOGGNWp279s3pdpZTKjn7b9hufL/2cJFeSx8/tiYFXKcCLIrLSGJMPWGGMmSUiGy45boGItPHA9ZRSKlsSEV75/RVOJ53m8Zse9/j5s5zwReQAcCD19iljzEagFHBpwldKKXUVkzZPYvXB1Qy7ZxghQZ6fCMGjdfjGmGigDrA0jd0NjTFrjDEzjDHVr3KO3saYGGNMTGxsrCfDU0opv+UWNwPmDqBS4Uo8eMODXrmGxxK+MSYv8DPwnIicvGT3SqCsiNQCPgcmXuk8IjJEROqJSL3IyDTn/1FKqWxn0qZJrDm0hv7N+nuldA8eSvjGmFBssh8pIhMu3S8iJ0UkPvX2dCDUGFPUE9dWSqlA5xY3A+YNoHKRyjxQ4wGvXSfLHyPGGAN8B2wUkf9e4ZjrgEMiIsaY+tgPmqNZvbZSSmUHv2z8hbWH1jKiwwivle7BM710GgNdgb+MMatTH+sHlAEQka+Be4HHjTEpwBngARERD1xbKaUCmsvtov/c/lQpUsWrpXvwTC+dhYC5xjFfAF9k9VpKKZXdDFszjA2xGxjfeTzBQcFevZaOtFVKKYecST5D/7n9qV+qPh2v7+j16/n1ildKKZWdfbn8S/ae3MvwDsOxzaHepSV8pZRywPGzx3l3wbu0qtiK5tHNfXJNTfhKKeWADxZ+QNzZON5r8Z7PrqkJXymlfGxH3A4+WfIJXWp2ofZ1tX12XU34SinlYy///jLBQcE+Ld2DJnyllPKpeTvnMX7DeF5p/Aql85f26bU14SullI+43C6e//V5ovJH8VKjl3x+fe2WqZRSPjJ09VBWHVzF6E6jiQiN8Pn1tYSvlFI+cOzMMfrO7kujqEbcX/1+R2LQhK+UUj7Qb3Y/jp05xv9a/88ng6zSoglfKaW8bOnepQxZMYRnGjxDretqORaHJnyllPKiFHcKj097nBL5SjCg+QBHY9FGW6WU8qKvln/FqoOrGHvvWPLnzu9oLFrCV0opL9l9Yjf95vTjjgp30LlaZ6fD0YSvlFLeICL839T/Q0QY3GawYw21F9IqHaWU8oIRa0cwc+tMPm31KdEFo50OB9ASvlJKedyh+EM89+tzNIpqxFP1n3I6nH9owldKKQ8SEZ6c/iTxSfF81+47goz/pNksR2KMiTLG/GGM2WiMWW+MeTaNY4wx5jNjzFZjzFpjzI1Zva5SSvmjkX+N5OeNP/NG8zeoWrSq0+FcxBN1+CnAiyKy0hiTD1hhjJklIhsuOOYuoFLq1gD4KvWnUkplG3tO7OGp6U/ROKox/2r0L6fDuUyWS/gickBEVqbePgVsBEpdclh7YJhYS4CCxpgSWb22Ukr5C7e46T6pOy5xMazDMIKDgp0O6TIerVwyxkQDdYCll+wqBey54P5eLv9QOHeO3saYGGNMTGxsrCfDU0opr/l86efM2TGHT+78hPKFyjsdTpo8lvCNMXmBn4HnROTkpbvTeIqkdR4RGSIi9USkXmRkpKfCU0opr1lzcA2v/P4KbSu3pUedHk6Hc0UeSfjGmFBssh8pIhPSOGQvEHXB/dLAfk9cWymlnHQ66TT3j7+fwuGF+a7dd34xwOpKPNFLxwDfARtF5L9XOGwy8Ehqb52bgRMiciCr11ZKKac9PeNpthzdwsiOI4nM49+1Ep7opdMY6Ar8ZYxZnfpYP6AMgIh8DUwHWgNbgQTgUQ9cVymlHDXqr1H8sPoHXm/6OreWu9XpcK4pywlfRBaSdh39hccI8GRWr6WUUv5i05FN/N/U/6NJmSb0b9bf6XDSxX+GgCmlVICIT4qn49iOhIeEM7rTaEKCAmNassCIUiml/ISI0HNyTzYf3cysrrMonb+00yGlmyZ8pZTKgM+Xfc7Y9WN5r8V73FbuNqfDyRCt0lFKqXSat3MeL/72Iu2qtOPlxi87HU6GacJXSql02Hl8J/f+dC8VC1dk2D3D/GoWzPQKvIiVUsrHTiedpv2Y9iS7kpn0wCQKhBVwOqRM0Tp8pZS6inOToq07vI7pD02ncpHKToeUaZrwlVLqKl6f8zrjN4zn45Yfc2fFO50OJ0u0Skcppa5g6OqhvLvwXXrd2IsXGr7gdDhZpglfKaXSMHfnXHpP6U3L8i35svWXfj0pWnppwldKqUtsiN1Ax7EdqVSkEuM6jyM0ONTpkDxCE75SSl1g38l9tBrRitwhuZn20DQKhhV0OiSP0UZbpZRKdeLsCe4aeRdxZ+OY330+0QWjnQ7JozThK6UUkJiSSIexHdh4ZCPTH5pOnRJ1nA7J4zThK6VyvBR3Cg9NeIg/dv7B8A7DaVmhpdMheYXW4SulcjQRoc/UPkzYOIFBdw6iS80uTofkNZrwlVI52iu/v8J3q77j9aav8+zNzzodDmfOgNvtnXNrwldK5VjvzH+HjxZ9xBP1nuCN5m84HQ4Ab78NNWvC2bOeP7cmfKVUjjRoySBe++M1utbsyuetP/eLgVWnT8NXX0GVKhAW5vnzeyThG2O+N8YcNsasu8L+5saYE8aY1albYCwAqZTKloasGMLzvz5Pp+s78X377/1mquOhQyEuDl7w0iwOnuqlMxT4Ahh2lWMWiEgbD11PKaUy5cfVP9Jnah9aV2rNqE6j/GY9WpcLPvkEGjSARo28cw2PfKyJyHzgmCfOpZRS3jJy7UgenfQoLcq3YHzn8eQKzuV0SP+YMgW2bYMXXwRv1S758ntMQ2PMGmPMDGNM9SsdZIzpbYyJMcbExMbG+jA8pVR2NnbdWB6Z+AjNo5sz6YFJhIeGOx3SRQYOhOho6NDBe9fwVcJfCZQVkVrA58DEKx0oIkNEpJ6I1IuMjPRReEqp7Gzc+nE8POFhmpRpwpQHpxARGuF0SBdZtgwWLoTnnoMQL9Yw+SThi8hJEYlPvT0dCDXGFPXFtZVSOduYdWN46OeHaBTViKkPTiVPrjxOh3SZjz6CAgXgsce8ex2fJHxjzHUmtc+TMaZ+6nWP+uLaSqmca9Rfo3h4wsM0LtOY6Q9PJ1/ufE6HdJmNG+Hnn+GppyCfl8PzyJcHY8xooDlQ1BizF/gPEAogIl8D9wKPG2NSgDPAAyIinri2UkqlZcTaEXSb2I2mZZv6bcke4N13ISLCVud4m0cSvog8eI39X2C7bSqllNd9v+p7ek7uya3lbmXyA5P9Ntlv2wajRtl+90V9UMntH6MNlFLKQ76O+Zoek3twR4U7/LpkD/DeexAaarti+oImfKVUtjFoySAen/Y4bSq3YeIDE/2u6+WFdu+GH3+EXr3guut8c01N+EqpbOGd+e/w/K/P0/H6jvx838+EhXhhMhoP+uADO8Dq5Zd9d01N+EqpgCYi9Jvdj9f+eI0uNbsw9t6xfjWCNi3bt8M330CPHhAV5bvr+sckEkoplQlucfP8zOf5bNln9L6xN1+1+cpvJkK7mv/8xw6wev11315XE75SKiC53C56TenFD6t/4Pmbn2fgHQP9Yorja/nrLxg5Ev71LyhZ0rfX1oSvlAo4Sa4kukzowk8bfmJAswH0b9Y/IJI9wGuvQf788Morvr+2JnylVEBJSE7g3nH3MmPrDAbeMZAXGnpp8ngvWLQIJk+2q1oVLuz762vCV0oFjBNnT9B2dFsW7l7IkDZD6FW3l9MhpZuILdUXKwbPOrR0riZ8pVRAiD0dS6uRrVh7aC2jO43m/hr3Ox1Shvz0k50R8+uvIW9eZ2LQhK+U8nt7Tuyh5fCW7Dqxi0kPTKJ1pdZOh5QhZ87YRtpataBnT+fi0ISvlPJrW45u4fZht3Mi8QS/dfmNW8re4nRIGTZw4PmRtcHBzsWhCV8p5bdWHVjFnSPuBGBut7nUKVHH4Ygybt8+O2dOx47QvLmzsfj/CAWlVI60YNcCmv/YnPDQcBY+tjAgkz3Av/9tFyj/6COnI9GEr5TyQ1O3TOWOEXdQMl9JFj66kMpFKjsdUqbMng0jRtj6+/LlnY5GE75Sys+MXDuSe8bcQ41iNVjw6AKiCvhwshkPOnMG+vSBihWhXz+no7G0Dl8p5Tc+W/oZz858lubRzZn0wCTy587vdEiZ9u67sHUr/P47hPvJLM2a8HOYxES7hua6dbYx6fBhuyUm2qlajbF9hIsXtwNEoqOhWjX7ddTJ3gUqexMRBswdwJvz3+SeqvcwutNov5/e+Go2bLDTH3ftCi1aOB3NeZrws7nTp2H+fJg1y9Ynrl9vG5DOyZMHIiMhLMyOBBSBU6fsh8CFx4WFQc2acMst0LSp/VmokO9fj8p+3OLm6elP87+Y//Fo7UcZ0nYIIUGBm5pcLruoSf78tjumP/HUIubfA22AwyJSI439BvgUaA0kAN1FZKUnrq0ul5wMv/5qG4smTYKzZyF3bmjSxPYYuOEGqFHDlt7zXGH1N7cbjh2z83avX2+3Zcvgiy/sP3FwsE387dpBhw5QtqxPX6LKJpJcSXSb2I0x68bwYsMX+ajlRwEzCdqVDBxo58wZPtwWpvyKiGR5A5oCNwLrrrC/NTADMMDNwNL0nLdu3bqi0u/wYZEBA0SKFbNl9SJFRJ58UmTWLJGEBM9c48wZkfnzRfr1E6le/dx3ApGmTUW++Ubk+HHPXEdlf/GJ8dJqRCthAPLBwg+cDscj1qwRyZVLpFMnEbfbmRiAGLlCTjV2f9YZY6KBqZJ2CX8wMFdERqfe3ww0F5EDVztnvXr1JCYmxiPxZWf79sE778APP9jSfOvWtndAq1Z2gWSPcrnsBffuhUOH2Lb+LGMXlODHmBpsOVaUiJBEHq4cw5M1F1CrxGHbIJAvHxQsaBfuLFECSpWytwO8JKcy79iZY7QZ1Yal+5YyuM1get7o4HwDHpKYCPXrw6FDds57p0r3xpgVIlIvrX2+qigrBey54P7e1McuS/jGmN5Ab4AyZcr4JLhAdfw4fPghDBoEKSnQrRu88AJcf70HTi4Cf/8NK1fa/961a2HzZti509YZpaoA9AP6YlgW3oxv3I8xYkMnvtnQmFuC/uRf7ve5m2kEcUnBIiLC9lerVMlOMFKrFtSpA6VL6wdBNrf/1H7uHHEnW45u4afOP9Hx+o5Oh+QRb7xh3yaTJ/thVU4qXyX8tN7BaX61EJEhwBCwJXxvBhWoRGxp/uWX4ehRePhheOstKFcuCyd1uWD1atuHbMECWLLEnhzsWmxVq9qk3LGj7bJTpsz5rjyFC2PCwmhgDA2AD4/Z+D77rDHtdk+hWlUXL/c6zsMNtxMSewD27IFt22yftTVrYMIE+6LAlv6bNLHb7bdDlSr6AZCNbD22lZbDW3Ik4QgzHp7BbeVuczokj5g1C95/Hx57DNq2dTqaq7hSXU9GNyCaK9fhDwYevOD+ZqDEtc6pdfiX27DB1peDSJMmIitXZuFkx4+LjBolcv/9IoULn6+Qr1pV5LHHbKX8qlUiZ89m6vRJSSIjRojccIM9bYUKIj/8IJKcfMmBp06JLFok8vnnIg88IBIVdT6WMmVEevYUmTLFNiCogLVy/0op9lExKfphUVm+b7nT4XjMvn0ikZEi1aqJxMc7Hc3V6/B9lfDv5uJG22XpOacm/PNSUkQ+/FAkNFSkUCGRb78VcbkycaKTJ0V+/FGkVSt7MrCtvN27i4wcKXLggMdjd7tFJk0SqVPnfOIfNeoa8W/fLjJ4sEjHjiL589sn5s1rP5wmThRJTPR4nMp75u6YK/nfyy9R/42SjbEbnQ7HY5KTRZo1E4mIEFm/3uloLK8nfGA0tj4+GVs/3wPoA/RJ3W+AL4FtwF9AvfScVxO+tWuXSPPm9q/VsaPIoUMZPIHLJfL777b0HBZmT1S2rMhLL4n8+af9NPGBc4m/Zk0bQq1aItOmpaM3Q2KiyIwZIr1726IU2G8kjz8usmKFL0JXWTBp0yTJ/VZuuf6L62X38d1Oh+NRr75q/x1//NHpSM7zSQnfG5smfFuYLVDAFm5/+CGDXb2OHBF5/32R8uXtn7pQIZEnnrBJ3qk+Y2I/f0aNOh/WrbdmIG8nJdlPiQs/vOrVs195Tp/2atwq435Y9YMEvxEsNw25SWJPxzodjkeNH2///R57zOlILqYJPwAlJ4u88sr5fLZtWwaevH69LQ2Hh8s/neRHjPC7OvDERJEvvhApWtSG2aWLyO6MFADj4kQ+++z8gIAiRWyRa/9+b4WsMuCjPz8SBiAth7WUU4mnnA7Ho1atstU4N9/sd28rTfiB5vBhW+oFkf/7vwy0mS5ZItK+vX1iWJhIr14if/3lzVA94vhxkX//WyR3bvsZ9frrth033dxukXnzRDp0EDHGtk307CmydavXYlZX5na75ZVZrwgDkM7jOsvZ5Mw1+vurgwdtv4LSpb3S5JVlmvADyPr1IuXK2eQ3dGg6n7RwoUiLFvJPtc2AASKxgff1eccO2yYLIiVLigwblomG6a1b7fDi3LlFgoJEHn5YZNMmb4Sr0pDsSpYek3oIA5A+U/pIiss37UO+kpAg0qiRLZhkqYecF2nCDxAzZ9oOKcWL28L6NcXEiNx1l/zT0+ajj2wvnAD355+2GgtEGjRI5+/iUvv320bpPHls4n/0UZGdOz0eqzrvTPIZuWfMPcIApP+c/uJ2sJ3IG5KTRdq1s18ix493Opor04QfAL75RiQ42PZg2bXrGgdv324bLc/1VvngA//oAOxBLpf9hnPddfZlduuWyar5Q4dEnn/elvhDQ0Wee07k6FFPh5vjnTh7QpoPbS4MQD5d8qnT4Xic220bZ0Hkyy+djubqNOH7Mbfb1sCALaxftYB+/LjIiy/a2ZnCw0Vee03kxAmfxeqEkydt43WuXLan0nvvZXIc2J49Ij162NJ+wYIiAwdmekCZutjBUwelztd1JOTNEBm5dqTT4XhF3772Pdq/v9ORXJsmfD+VnGw705wrwSYlXeFAl0vku+9stY0xtqixd68vQ3Xc33+fb4+uUMF2V81UjcHatSJ33mlPVLmyrUdTmbb92Hap+FlFiXgnQmb8PcPpcLziP/+x/y69ezvamzndNOH7ocREkXvvtX+Bvn2v8o+0YoVI/fr2wMaN/belyEd++80OYQeR22/PQiek6dNFKlWyJ7rnHq3fz4Q1B9dIiY9LSKH3C8mi3YucDscrziX7Rx/N5Mh2B2jC9zOnT59va/344yscdPKkrW8OCrIV2cOHB0bxwgeSkmz3+4IF7a/niScy2Snp7FmRd9+1HaojIuwf47KJflRa5u+cLwXeKyClBpaSdYfWOR2Ox7ndgZnsRTTh+5WTJ+04KGPsVDFpmjLFdvI1xmazuDhfhhgwYmNtD8zgYJv8//vfTE6xs2uXSJs29u1Qu7bt/aSuaPKmyRL2dphU/ryy7IzLft+MXC6RZ54JzGQvognfbxw/LtKwoU1Qo0alccCRI3a4KYjUqCGyeLHPYwxE69aJ3HGH/bVVrCgyYUImvgy53bavXYkS9g/Ut6826qbh+5XfS/AbwVJvSD05HH/Y6XA8LjHxfAe4558PvGQvognfL8TF2ar4kBCbkC7zyy+2UTYkxH6X1NkgM2zGjPP1+7fcksn++3FxtlgHItdfL7J0qafDDEhut1s+WPiBMAC5fdjtcvJs4I/3uFRcnG0XAtvTOVBrUDXhO+zYMZG6dW038IkTL9l5/LjtonOuOmHNGidCzDaSk0W++ur8ur6dO9sePhk2Y4atVgsOth/AV+xClf253C55YeYLwgDkgfEPSGJK9iuMbNpkO22FhNhJCgOZJnwHxcXZZJ8rl62av8jcuXZSjuBg26deS/Uec/Kk7TMdEWHfxH36ZGLgVlycSNeu8s8Mdhuzzzzu6ZWYkigP//ywMAB5evrT4nIHYB3HNcycaWekLVrUTskU6DThO+T4cVuNExp6SbJPShLp1882ylasmMm6B5UeBw7Yht2QEDtW7eWXM9GjZ/x4OxNnRIQdEh2o3/Uz6FTiKblz+J3CAOSd+e9ku6kSUlLsoMegIDvCfccOpyPyDE34Djhxwk6dGhJiF/34x7Zt5/vVP/ZYBqeFVJm1daudR80YO2L3tddsVVu67dt3voL33nsz+OTAczj+sNw05CYJeiNIvl3xrdPheNy+fecXFeraNXu9DTXh+1h8vF1vNjj4kgbacePs7GgFC9rbyufWr7f1+iCSL5+dPv/IkXQ+2eWyrXkhIXat3Wzai2rbsW1S6bNKEvZ2mEzaNOnaTwgwEybYhdMiIjIwI20A0YTvQwkJdqbioCCRsWNTHzxzxi7HB7bYr6M6HbdmjU3850r8L71kS33psmyZSHS0Tfwff5ytqnhW7l8pxT8qLoXeLyR/7v7T6XA86sgRkQcftG/DOnX8Zw1aT9OE7yOJiSKtW9skMmxY6oPbtp1fvftf/8rRvT380bp1Ig89ZD+gc+Wy66aka/r8uDi7wDDYOXOzweC4WdtmSb5380nUf6Nkw+ENTofjMW63HfdSvLj9jH7jjez9NvTFIuatgM3AVuDfaexvDpwAVqdu/dNz3kBK+MnJ59//Q4akPjhliq2+KVhQZPJkR+NTV7dtmx3UfG6Z3DZtRObMuUbh3e0W+fRTm0XKl7fr3gWoEWtGSMibIVLjfzVkz4k9TofjMevXn189rm7dgP4TpZtXEz4QDGwDygO5gDVAtUuOaQ5Mzei5AyXhu1y2QRBEBg0S2/z/2mv2gRtvtPPXq4Bw6JDtuREZaf98N9xgp8C46nIDixaJlCplPy0CrBP3hQOqmg9tLnFn4pwOySMOHTrfO6tQITs2IyV7Lb51Rd5O+A2BXy+43xfoe8kx2Tbhu93npzh+5x2xvTfOzYzWo4f/rXCs0uXMGZFvv7Vj4cD2037mmavMznnokMhtt9mDn3wyIMZUpLhS5MlpTwoDkPt/uj9brD174oStssmb13aa6NPHrhGdk3g74d8LfHvB/a7AF5cc0xw4mlr6nwFUv8r5egMxQEyZMmW8/svJCrdb5IUX5J8pjmXtWjtZe2joVWZGU4HE7bZLBj/4oK3jP9fu/s03dpzFRZKTbesv2G5aBw86EnN6nE46Le1HtxcGIC/++mLAD6iKi7OJvlAh++vv2DHnLmXs7YTfOY2E//klx+QH8qbebg38nZ5z+3sJv39/+xt8+mkR988T7PqpJUrYr/gq24mNtQtlVa1q/+5hYfaDYPr0SxoBR4+2o7xKlRJZvtyxeK/kcPxhufnbm8UMMPLZks+cDidLdu60n7H588s/7efLljkdlbMcr9JJ4zk7gaLXOrc/J/wPPrC/vccedYur/wB7p379DPTtU4HK7baDox9//HyJsmhRW30wZ07qlPqrV9u++mFhV5ga1Rmbj2yWCp9WkLC3w+TnDT87HU6muFwis2fb8W9BQbbq5r77ckaDbHp4O+GHANuBchc02la/5JjrAJN6uz6w+9z9q23+mvC//NL+5u7vlCwpne6zdx55ROvrc6CzZ+2EeA88YAfynEv+PXqITBkeJwmNWpyv83N4rt0FuxZI4Q8KS+SHkbJ4T+ANGtu1y7aTlS9vf6WFCtn1jnfvdjoy/+KLbpmtgS2pvXVeTX2sD9An9fZTwPrUD4MlQKP0nNcfE/7Qofa31rZlgiTVqW873WezwTcqc+Lj7bQ7Dz10voohPNwtbcqskf/RR7a27HON7j7eM2rtKMn9Vm6p/Hll2Xp0qyMxZMaBA7aA1aSJ/X2C7WY5cqQd5Kgud7WEf67U7Zfq1asnMTExTofxj3Hj4MEHocVNJ5i8qzZh8Udg9Gho08bp0JSfSUyE+fNh6lSYMkXYscMAUD7XHm6/txDN2+SlWTMoWdK7cYgIb89/m/5z+9O0bFMm3DeBIhFFvHvRLBCB9eth2jSYOBGWLrWPVasGDz0EDzwAFSo4HaV/M8asEJF6ae7ThJ8+U6ZAx45wc6UjzNxRlTzF89p3c40aToem/JwIbN0Kv32ynt++2clc1y2clPwAVKwIjRrZrWFDm9hCQjxz3cSURHpN6cXwtcPpWrMr37T9htwhuT1zcg8RgR07YMECmDMHZs2CAwfsvrp1oX17uOce+zYzxtFQA4Ym/Cz67Tdo21aoWewQs/dWIX+D62HSJChe3OnQVKBZtw7X3e1YfbgEcx8YzPxjNVi8GGJj7e7wcKhTxya7WrXsVr26fTwjYk/H0mFsB/7c8ydvNn+T15q+hvGDjHnyJKxaBcuW2W3xYti3z+4rUgRatoQ77rBbqVLOxhqoNOFnwbx5cNddQuWIvcw5WovCnW+HH3/M+DtQqXMOHoR27SAmBgYORJ59ju07DIsXw4oV9uFVq+D0aXu4MVCuHFx/vd0qVbLfDCpUsEnx0m8E6w6vo+3othyMP8iP9/zIfdXv8+nLE4Fjx+Dvv2HzZrutXw9r18LOneePK1cOGjSAW26Bpk3tt5ugIJ+Gmi1pws+kxYuhZUuhDHuYd7oukf/uCe+8o/+VKusSEqBrV5gwAZ56CgYNguDgf3a73bB9u02Sa9fCxo1227LFtg+cExxsk35UlP15tsxUfs37EOHBeXm90iQalb2JggWhQAG7hYdnrmpEBM6cgePH4cQJOHoUjhyx30wOHLCl9P37Ydcum9RPnTr/3JAQ+yFVqxbUrGl/3nQTREZm8nenrkoTfiYsWwYtb3dTLHEv81MaUeKr/tC7tyOxqGzK7YaXX4aBA22Jf9QoyJPnqk9xuWxy3boVtm2D3bttkt21W9hQ6H2O1HoVDtwIYybCydKXPT8oCCIi7JY7N4SGQq5cF5dhXC5ITrbb2bM20Sck2HCvJDLSNkBHRdmSe7ly9ltIlSr2dmhoJn9HKsM04WfQypXQormLQgl7mZf7TqJ+HgStWvk8DpVDfPEFPPusrbifOhWKFcvQ0xOSE+g5uSej143mwRoP8ult3xF/PPyfEvi5UvmJE7aa6PRpm8CTks5v59KAiC2Rn9vCw+2HQ3g45Mt3/ptCkSI2yRctasPN7V9twTna1RK+h/oDZB9r1kDLW5PJH3+QOUU7E/XbGKhd2+mwVHb21FNQpoztc9iwIcycaetA0mFH3A46juvImoNreL/F+7zc+GWMMUQWsiVrpS6kldEXWLsWWjRJJOLkQf4o34Po5T9psle+0a4d/PGH7cbSqBEsWXLNp8zaNot639Rj5/GdTHtoGq80ecUveuIo/6UJP9XatXBbwzOExcfyR72XKb98LJQt63RYKidp0AAWLbJ1JrfdZgd/pMEtbt5b8B6tRraiZL6SLO+1nLsq3eXjYFUg0oQP/LXGTYub4wlLOMrcO9+n4oIfoFAhp8NSOVGlSjbpV69uRxx9++1Fu+POxHHPmHvoN6cfnat1ZnGPxVQsXNGZWFXAyfEJf/XyZG5tcJrcZ44zt8t3VJz2KYSFOR2WysmKFbPVOy1bQq9e8OabIMLKAyup9009ZmydwWetPmN0p9HkzZXX6WhVAMnRjbYx805zx+0u8qYcZ86L06n4UX8dv638Q968tkqnVy/kP//hyxOzeLHQMorlKcb87vNpGNXQ6QhVAMqxCX/pjGPc2TaUQq6jzPlwBeX+1cfpkK4qPimePSf2sPfkXo4kHOHYmWPEnY3jbMpZklxJJLmSCAkKIVdwLnIF56JgWEGKRhSlaERRSucvTbmC5ciT6+p9vJWfCQ0l7n//pWeZxUwIXkibuOsY+uQSihTSOQdU5uTIhD9/7AHufjAfxTnEnO93UubRe50OCbAzG+4/tZ81h9aw9tBaNh3ZxOajm9lydAvHzhxL8znBJphcwbkIDQ7F5XaR5Eoi2Z2c5rGREZFULVqVmsVrUrN4TeqVrEfN4jUJCcqR/wZ+b/6u+XSZ0IUDoQf4OFc7XhgwGbOyi51GskABp8NTASjHvdN//Wo7HZ4oQXTQbn7/5RQl27VwLJZjZ46xZO8Slu1bxrJ9y1i+fzlHEo78s79kvpJUKVKFztU6E10wmqj8UUQViCIyIpLC4YUpFF6IXMG5LjuvW9wcP3ucIwlHOJJwhN0ndrMjbgfb47az4cgGhq0ZxqkkO/Y9IjSCBqUa0KxsM1pWaEn9UvX1A8Bhya5k3pj3Bu8ueJcKhSuw6LFF3FTqJogaBd26QbNmtq/+ddc5HaoKMDlqpO2kd9Zx32uVqBb6N7/NCSWySRWPnftaRIRdJ3axYNcCFuxewJ97/mRD7AYADIbqxapzU8mbqHNdHWpfV5uaxWtSIMw7pTgRYefxnSzdt5RFexbx554/WXVgFYJQIHcBWlZoyT1V7qF1pdYUCtfeSr60IXYDj/zyCCsOrOCx2o/x6V2fXtww++uv0KmTnan1t990cnh1GZ1aARjxXAzdP61NvbD1zFhWhEI3XD7PiCeJCH8f+5t5O+cxb9c85u+az56TewAoGFaQRlGNaBzVmIalG1KvZD3y5c7n1Xiu5WjCUebsmMNv235j2t/TOBB/gJCgEG6NvpUHazxIh+s7UDCsoKMxZmcut4tBSwbx6pxXyZc7H1/f/TWdqnVK++ClS+Huu+3MaTNn2vmUlUqV4xP+l10W89TIhtyWfzkT15QnX7TnV/xxuV38dfgvFu5eyPxd85m/az6HTh8CoHie4jQt25RmZZvRtGxTqherTpDx3x6xbnGzfN9yftn0Cz9t+IntcdvJFZyL1pVa071Wd1pXak1osM6G5SkbYzfSc0pPFu1ZRPsq7RncZjDF815jrYVNm+yk8SdOwOTJtppHKXJwwhe38F7rBbz6a1PaF1vEmPU1CSvqmX7LRxKOsGzfMpbuXcrivYtZsnfJP/XiZQqUoWnZptxS5haalW1G5SKVA3bIu4gQsz+G0etGM+qvURw6fYiiEUXpckMXet7Yk+rFqjsdYsBKciXx4Z8f8tb8t8ibKy+D7hxEl5pd0v+/snevTfrbt8OYMXaglsrxvJ7wjTGtgE+BYOBbEXn/kv0mdX9rIAHoLiIrr3XerCR8d4qbl25ewCcrmtGl7AK+X9+A0DyXN3Bei8vtYsfxHWyI3cDaQ2tZdXAVKw+sZOfxnQAEmSBqFKtB46jGNCnThMZRjSlbMHtOyZDiTuHXrb8ydM1QJm2aRLI7mYalG9K7bm/uq34fEaERTocYMObtnMcT059gQ+wG7q9+P5/d9RnF8mRslkzATkx/992wfDkMGQI9eng+WBVQvJrwjTHBwBagJbAXWA48KCIbLjimNfA0NuE3AD4VkQbXOndmE35yQjI9ay5l2LYmPFNzLp+saEpQyOVVKCLC6eTTHDtzjIPxB9l3ch/7Tu1j5/GdbIvbxrZj2/j72N+cTTn7z3MqFa5EnRJ1uPG6G2lQugH1StbLkaMdY0/HMmzNML5Z+Q2bj26mYFhButfqTp96fahS1HeN4YHmYPxB/jXrX4xYO4LogtF8ftfntKncJmsnPX0a7r3X1ue//76dYz9Av1GqrPN2wm8IDBCRO1Pv9wUQkfcuOGYwMFdERqfe3ww0F5EDVzt3ZhL+mWNnKP/8XRx0h1Gx8FHK1SuES1wku5JJdidzJvkM8UnxxCfFE3c2jiRX0mXnCAsJo3yh8lQoVIFKhStRvVh1qkVWo1pkNfLnzp+heLI7EWH+rvl8FfMVEzZOINmdTItyLXjipidoV6WddvFMdSb5DJ8s+YT3Fr5HkiuJlxu9TN9b+nruW1FSEnTvDqNHw4svwocf6spsOZS358MvBey54P5ebCn+WseUAi5L+MaY3kBvgDJlymQ4GHELCblOEl3kMEUq5udU0ilCgkIIDQolPDSc4nmKkydXHvKE5vlnNGqR8CIUz1ucUvlKUSp/KYpGFPXrRlV/YoyhWXQzmkU341D8Ib5b9R1fx3xNp3GdKJWvFL3r9qbXjb0oka+E06E6wi1uxqwbQ7/Z/dh1Yhftq7Tno5YfUalI+ua7T7dcuWDECLsyycCBdv3Bb77RpabUxUQkSxvQGVtvf+5+V+DzS46ZBjS54P5soO61zl23bl3JjJTElEw9T3lGiitFJm6cKHcOv1MYgIS8GSL3jrtXft/2u7jcLqfD8wm32y2TNk2SG/53gzAAqf11bZmzfY4vLizyxhsiINKmjUhCgvevqfwKECNXyKmeKOHvBaIuuF8a2J+JYzwmOFfwtQ9SXhMcFEz7qu1pX7U9W49t5euYr/lh9Q+M3zCeioUr0vvG3nSr3S1zjZR+zi1uJm+ezDsL3iFmfwyVCldiTKcxdK7e2TffGo2B/v3t+oNPPml78UyZAgULev/ayu95og4/BNto2wLYh220fUhE1l9wzN3AU5xvtP1MROpf69xOLmKuPOtsylnGbxjP4BWDWbh7ISFBIbSr0o4edXpwR4U7Ar6u/2zKWcauG8uHiz5kQ+wGKhSqQN8mfXmk1iPOjVkYNw66dIGqVe0I3RI5s1otp/FFt8zWwCBst8zvReQdY0wfABH5OrVb5hdAK2y3zEdF5JqZXBN+9rQxdiPfrfqOH9f8yJGEIxTPU5yHbniIrjW7Uvu62gE1ZmHvyb0MjhnM4BWDiU2I5YZiN9C3SV86V+/sHx9is2ZBhw62xP/bb+leK1cFrhw78Er5tyRXEtO2TGP42uFM3TKVZHcylYtU5r5q93Ff9fuoUayGXyb/sylnmbRpEj+s/oFZ22chIrSr0o5nGjzDrdG3+l/My5fDXXfZqRhmzIAbb3Q6IuVFmvCV3zuacJTxG8YzbsM45u6ci1vclCtYjnZV2tG2clsal2lMWIhzK5ElJCcwc+tMxm8Yz9QtUzmVdIoyBcrQrVY3utfuTvlC5R2LLV02b7b1+XFxdnrl225zOiLlJZrwVUA5FH+IiZsmMmXLFH7f/juJrkTCQsJoUqYJLcq1oHFUY+qWrOvVkb1JriTWHFzDnB1zmLV9Fgt2LyDJlUSR8CJ0qNqB+2vcz23lbgus7rv79sGdd8Lff8PIkXawlsp2NOGrgHU66TRzdsxh9o7ZzN4xm3WH1wF24ZdzC7lUj6xO9WLVKV+oPFH5ozK0spfL7eLw6cNsObqFjUc2sjF2IzEHYlixfwWJrkQAbih2Ay3Lt+TuynfTtGxT/6ibz6xjx6BtW1i8GL78Eh5/3OmIlIcFbMLPVy6f1P1P3Yseu6/6fTxx0xMkJCfQemTry57TvXZ3utfuzpGEI9w77vISzOP1Huf+Gvez58Qeuv7S9bL9LzZ8kbZV2rL5yGb+b+r/Xbb/taavcXv521l9cDXPzXzusv3vtniXRlGNWLRnEf1m97ts/6BWg6h9XW1+3/47b89/+7L9g9sMpkrRKkzZPIWBiwdetn94h+FEFYhi7LqxfBXz1WX7x983nqIRRRm6eihDVw+9bP/0h6cTERrB/5b/j3Hrx122f273uQB8vOhjpm6ZetG+8NBwZjw8A4C35r3F7B2zL9pfJKIIP9/3MwB9f+/L4r2LL9pfOn9pRnQcAcBzM59j9cHVF+2vXKQyQ9oOAaD3lN5sObrlov21r6vNq7e8ytJ9S+n7e1/2ndrH6aTTJLkvHi1dJLwIia5EDIaQoBCCTBAGQ7E8xShXqBzxSfGsOrjqn+UhL5QnNA+1r6vNruO7yJ87PwXCCvyzyEy2+d/bMJW3v+8GR49B2bIQHQ3o/961/vcGtRoEQJcJXdh7cu9F+xuWbsh7t9vJBTqN68TRhKMX7W9RrgWvN3sdgLtG3sWZ5DMX7W9TuQ0vNXoJgOZDm3OpjPzvReaJ9OpIW6V8JjJPJG0qt2HMujEUibDTXKe4UzidfJqo/FE0LN2Q3Sd2M2XLFE4nnybZnWwHnSDEJsQSHhpO3lx5yR2cm/CQcHKH5CZXcC7CQ8JpW6Utb936FkEmKM03XbYRFgbVa8CWLbBrl52WQXvv5Ah+XcLXKh2lvEgEXn8d3nnHVvOMGQMROuNpoLtalU4AtTgppTzKGHj7bfjiC5g6FVq0sHPwqGxLE75SOd2TT8L48bBqFTRpAjt2OB2R8hJN+Eop6NjRjso9dAgaNoQVK5yOSHmBJnyllHXLLfDnn5A7t10jd8YMpyNSHqYJXyl1XrVqto9+pUq2Iffbb52OSHmQJnyl1MVKloT58+H226FXL3j1VXC7nY5KeYAmfKXU5fLls/Po9+oF775rp1lOTHQ6KpVFOvBKKZW20FAYPBgqVIB//xv27IFffoGiRZ2OTGWSlvCVUldmDLzyCowda6dZvvlm2LTJ6ahUJmnCV0pd2333wdy5cPKk7bY5Z47TEalM0ISvlEqfm2+GZcugVCk7zfLgwU5HpDJIE75SKv2io2HRIruYSp8+8MwzkJLidFQqnTThK6UyJn9+mDwZXngBPv8cWre28+wrv5elhG+MKWyMmWWM+Tv1Z6ErHLfTGPOXMWa1MUanv1Qq0AUHw8CB8N13tm6/fn1Yv97pqNQ1ZLWE/29gtohUAman3r+SW0Wk9pWm7VRKBaDHHrMJPz7e1vFPnOh0ROoqsprw2wM/pt7+Ebgni+dTSgWaRo0gJgaqVoUOHeC118DlcjoqlYasJvziInIAIPVnsSscJ8BvxpgVxpjeVzuhMaa3MSbGGBMTGxubxfCUUj5RujQsWGBL/O+8A23aaL2+H7pmwjfG/G6MWZfG1j4D12ksIjcCdwFPGmOaXulAERkiIvVEpF5kZGQGLqGUclRYmJ1sbfBgmD0b6tbVaZb9zDUTvojcLiI10tgmAYeMMSUAUn8evsI59qf+PAz8AtT33EtQSvkNY6B3b1vad7lsdc/gwXY5ReW4rFbpTAa6pd7uBky69ABjTB5jTL5zt4E7gHVZvK5Syp81aAArV8Ktt9r++o88Yht2laOymvDfB1oaY/4GWqbexxhT0hgzPfWY4sBCY8waYBkwTURmZvG6Sil/V7QoTJ8Ob7wBo0bZKp7Vq52OKkcz4sdfterVqycxMdptX6mAN3cuPPSQbcgdOBCeeMJW/yiPM8asuFL3dx1pq5TyvubNYc0auO02eOopaNcODqfZ5Ke8SBO+Uso3IiNh6lT49FO7YHrNmrpuro9pwldK+U5QkJ1wbfly+wHQurVt1NUGXZ/QhK+U8r0bbrBJ/6WXYMgQW9qfP9/pqLI9TfhKKWeEhcFHH8G8ebYBt3lzePppOHXK6ciyLU34Siln3XKLbdB9+mn48kuoUQN+/dXpqLIlTfhKKeflzWsbcxcuhIgIaNXKduM8eNDpyLIVTfhKKf/RqJEdnPXGGzBhAlSpAl98oatqeYgmfKWUf8mdG/r3h7/+slM0PP20HaU7d67TkQU8TfhKKf9UqZKtyx8/Hk6csPPydO4M27Y5HVnA0oSvlPJfxkCnTrBxI7z5pp2b5/rr4dlnQdfLyDBN+Eop/xceDq+/Dlu3wqOP2nr9ChXgP/+B48edji5gaMJXSgWOEiXs/Prr1sEdd9hSf7ly8NZbmvjTQRO+UirwXH+9rdtftQqaNbONvGXKwCuvwIEDTkfntzThK6UCV+3aMHGiTfx33w0ffwzR0dCtm12ARV1EE75SKvDVrg2jR8OWLdCrF/z8s+3K2aQJDBsGCQlOR+gXNOErpbKPChVsg+6+ffDJJ3bO/W7dbN3/44/Dn3+C2+10lI7RhK+Uyn4KFIDnnoPNm+2ArfbtYehQW+IvV87W9S9dmuOSvyZ8pVT2ZYxt1B02DA4dguHD7eRs//0v3HwzlCwJPXvCuHH+068/MdFWTXlBlta0NcZ0BgYA1wP1RSTNBWiNMa2AT4Fg4FsReT8959c1bZVSXhEXBzNnwuTJdtWtEyfs4zVrQuPG9sOgQQM72jfIi+ViEdi/3zY6L1kCCxbYbx5FisDevZla9/dqa9pmNeFfD7iBwcBLaSV8Y0wwsAVoCewFlgMPisiGa51fE75SyutSUmDFCpg9G+bMgWXLzs/JHxEB1arZbwUVK9oeQNHRtk0gMtLO8nmtpJyUZL897N9vtx07bFXTli12PMG5tX2Dg+HGG+100bfcYtf9zcSHzdUSfkiGz3YBEdmYeoGrHVYf2Coi21OPHQO0B66Z8JVSyutCQmxpvkED6NcPXC7YtMmWtP/6C9avt3P6DB16+XNz5YJ8+exI4PBwm6BdLrslJNhvDmfPXv68ggXtTKCtW9skf+ONUKuW/QDx5kv16tmtUsCeC+7vBRpc6WBjTG+gN0CZMmW8G5lSSl0qOBiqV7fbhRISYPdu2LnTztMfG2u3+Hg4c8ZuIvb5wcH2A6BAAbsVLQqlStk2gzJl7P1MVNdk1TUTvjHmd+C6NHa9KiKT0nGNtF7VFeuRRGQIMARslU46zq+UUt4XEQFVq9otQF0z4YvI7Vm8xl4g6oL7pYH9WTynUkqpDPJFt8zlQCVjTDljTC7gAWCyD66rlFLqAllK+MaYDsaYvUBDYJox5tfUx0saY6YDiEgK8BTwK7ARGCci67MWtlJKqYzKai+dX4Bf0nh8P9D6gvvTgelZuZZSSqms0ZG2SimVQ2jCV0qpHEITvlJK5RCa8JVSKofI0lw63maMiQV2ZfLpRYEjHgwnEOhrzv5y2usFfc0ZVVZEItPa4dcJPyuMMTFXmkAou9LXnP3ltNcL+po9Sat0lFIqh9CEr5RSOUR2TvhDnA7AAfqas7+c9npBX7PHZNs6fKWUUhfLziV8pZRSF9CEr5RSOUS2S/jGmFbGmM3GmK3GmH87HY8vGGO+N8YcNsasczoWXzDGRBlj/jDGbDTGrDfGPOt0TN5mjAkzxiwzxqxJfc1vOB2Trxhjgo0xq4wxU52OxReMMTuNMX8ZY1YbYzy6qHe2qsPPyoLpgcwY0xSIB4aJSA2n4/E2Y0wJoISIrDTG5ANWAPdk57+zsQtH5xGReGNMKLAQeFZEljgcmtcZY14A6gH5RaSN0/F4mzFmJ1BPRDw+2Cy7lfD/WTBdRJKAcwumZ2siMh845nQcviIiB0RkZertU9h1Fko5G5V3iRWfejc0dcs+pbUrMMaUBu4GvnU6luwguyX8tBZMz9aJIKczxkQDdYClDofidalVG6uBw8AsEcn2rxkYBLwMuB2Ow5cE+M0Ys8IY09uTJ85uCT9DC6arwGaMyQv8DDwnIiedjsfbRMQlIrWx60LXN8Zk6+o7Y0wb4LCIrHA6Fh9rLCI3AncBT6ZW2XpEdkv4umB6DpFaj/0zMFJEJjgdjy+JyHFgLtDK2Ui8rjHQLrVOewxwmzFmhLMheV/qioGIyGHsioL1PXXu7JbwdcH0HCC1AfM7YKOI/NfpeHzBGBNpjCmYejscuB3Y5GhQXiYifUWktIhEY9/Lc0Ski8NheZUxJk9qRwSMMXmAOwCP9b7LVgk/py6YbowZDSwGqhhj9hpjejgdk5c1BrpiS3yrU7fW13pSgCsB/GGMWYst2MwSkRzRTTGHKQ4sNMasAZYB00RkpqdOnq26ZSqllLqybFXCV0opdWWa8JVSKofQhK+UUjmEJnyllMohNOErpVQOoQlfKaVyCE34SimVQ/w/l3gd2N/VcY4AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Do some vector computations.\n",
    "y1, y2 = x.sin(), x ** x.cos()\n",
    "y3 = y2 - y1\n",
    "y4 = y3.min()\n",
    "\n",
    "# Print and plot some answers.\n",
    "print(f'The shape of x is {x.shape}')\n",
    "print(f'The shape of y1=x.sin() is {y1.shape}')\n",
    "print(f'The shape of y2=x ** x.cos() is {y2.shape}')\n",
    "print(f'The shape of y3=y2 - y1 is {y3.shape}')\n",
    "print(f'The shape of y4=y3.min() is {y4.shape}, a zero-d scalar')\n",
    "\n",
    "plt.plot(x, y1, 'red', x, y2, 'blue', x, y3, 'green')\n",
    "plt.axhline(y4, color='green', linestyle='--')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise\n",
    "\n",
    "Plot y3 clamped between 0.0 and 1.0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYEAAAD4CAYAAAAKA1qZAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAJYUlEQVR4nO3cW4jmd33H8c+32YYSWrFmt9YmadfWC6lQYhxCL1IIVUTSoC30UtrihQqKPVDbYBCk0IvEirbggSDYSBVvekQsbQ2FFoqR2ZxEo00itdUmTdILtbRFbL+9mP+WcZ3ZnXnm8Mzk+3rBn3nmf3jy+/KHvHf+z+xWdweAmb5n3QsAYH1EAGAwEQAYTAQABhMBgMHOrHsB+3H27Nk+f/78upcBcKpcuHDh2e4+t9OxUxWB8+fPZ3Nzc93LADhVquorux3zOAhgMBEAGEwEAAYTAYDBRABgMBEAGEwEAAYTAYDBRABgMBEAGEwEAAYTAYDBRABgMBEAGEwEAAYTAYDBRABgMBEAGEwEAAYTAYDBRABgMBEAGEwEAAYTAYDBRABgMBEAGEwEAAYTAYDBRABgMBEAGEwEAAYTAYDBRABgMBEAGEwEAAYTAYDBRABgMBEAGEwEAAYTAYDBRABgMBEAGEwEAAYTAYDBRABgMBEAGEwEAAYTAYDBRABgMBEAGEwEAAYTAYDBRABgMBEAGEwEAAYTAYDBRABgMBEAGEwEAAYTAYDBRABgMBEAGEwEAAYTAYDBRABgMBEAGEwEAAYTAYDBRABgMBEAGEwEAAYTAYDBRABgMBEAGEwEAAYTAYDBRABgMBEAGEwEAAYTAYDBRABgMBEAGEwEAAYTAYDBRABgMBEAGEwEAAYTAYDBRABgMBEAGEwEAAYTAYDBRABgMBEAGEwEAAYTAYDBRABgMBEAGEwEAAYTAYDBRABgMBEAGEwEAAYTAYDBLhuBqrq2qh5atqeq6mvbvv/Rqvrzqnqsqp6oqt+vqquX626tqq9X1YNV9aWq+ruquv2S935jVX1x2T5bVbcc5aAAfLczlzvY3f+e5MYkqap3JfmP7v69qqok9yf5YHe/rqquSnJPkt9N8vbl8r/v7tuXa29M8mdV9V/dfd8ShDcluaW7n62qm5bjN3f3U4c+JQA7WvVx0M8m+e/u/kiSdPf/JPn1JG+oqmsuPbm7H0ryO0neuuz67SRv7+5nl+MPJLk3yVtWXA8AK1g1Ai9LcmH7ju7+RpJ/TvKSXa55IMlLd7s+yeay/zssj402q2rzmWeeWXG5AOxk1QhUkt7H/ovH9v2e3X1Pd29098a5c+f2t0oALmvVCHw+ycb2HVX1vCQ3JHlil2tenuTR5fUXkrzikuM3LfsBOCarRuC+JNdU1S8lyfLB8HuS/GF3/+elJ1fVTyV5Z5L3L7vuTnJXVV27HL8xya8k+cCK6wFgBZf97aDddHdX1S8k+UBVvTNbMflUkndsO+1nqurBJNckeTrJ27r7vuX6v6iq65L8Q1V1km8meX13P3mAWQDYp+re7RH+ybOxsdGbm5vrXgbAqVJVF7p7Y6dj/sYwwGAiADCYCAAMJgIAg4kAwGAiADCYCAAMJgIAg4kAwGAiADCYCAAMJgIAg4kAwGAiADCYCAAMJgIAg4kAwGAiADCYCAAMJgIAg4kAwGAiADCYCAAMJgIAg4kAwGAiADCYCAAMJgIAg4kAwGAiADCYCAAMJgIAg4kAwGAiADCYCAAMJgIAg4kAwGAiADCYCAAMJgIAg4kAwGAiADCYCAAMJgIAg4kAwGAiADCYCAAMJgIAg4kAwGAiADCYCAAMJgIAg4kAwGAiADCYCAAMJgIAg4kAwGAiADCYCAAMJgIAg4kAwGAiADCYCAAMJgIAg4kAwGAiADCYCAAMJgIAg4kAwGAiADCYCAAMJgIAg4kAwGAiADCYCAAMJgIAg4kAwGAiADCYCAAMJgIAg4kAwGAiADCYCAAMJgIAg4kAwGAiADCYCAAMJgIAg4kAwGAiADCYCAAMJgIAg4kAwGAiADCYCAAMJgIAg4kAwGAiADCYCAAMJgIAg4kAwGAiADCYCAAMVt297jXsWVU9k+Qr617HCs4meXbdizhmZp7BzKfDj3X3uZ0OnKoInFZVtdndG+tex3Ey8wxmPv08DgIYTAQABhOB43HPuhewBmaewcynnM8EAAbzkwDAYCIAMJgIHJKqekFV/U1VPbZ8/cFdzntNVX2pqh6vqjt2OP6bVdVVdfboV30wB525qt5dVV+sqkeq6k+r6vnHtvh92MM9q6r6g+X4I1V1016vPalWnbmqbqiqv62qR6vq81X1q8e/+tUc5D4vx6+qqger6pPHt+pD0N22Q9iS3J3kjuX1HUnu2uGcq5I8keTHk1yd5OEkP7nt+A1J/ipbfyHu7LpnOuqZk7w6yZnl9V07Xb/u7Ur3bDnntiR/maSS/HSS+/d67UncDjjzi5LctLz+gST/+Fyfedvx30jy8SSfXPc8+9n8JHB4Xpfk3uX1vUl+fodzbk7yeHd/ubu/leQTy3UXvTfJbyU5LZ/WH2jm7v7r7v72ct5nklx/tMtdyZXuWZbvP9pbPpPk+VX1oj1eexKtPHN3P9ndDyRJd38zyaNJrjvOxa/oIPc5VXV9kp9L8uHjXPRhEIHD88LufjJJlq8/tMM51yX5l23ff3XZl6p6bZKvdffDR73QQ3SgmS/xhmz9Keuk2cv6dztnr7OfNAeZ+f9V1fkkL09y/+Ev8dAddOb3ZesPcP97ROs7MmfWvYDTpKo+neSHdzh0517fYod9XVXXLO/x6lXXdlSOauZL/ht3Jvl2ko/tb3XH4orrv8w5e7n2JDrIzFsHq74/yR8n+bXu/sYhru2orDxzVd2e5OnuvlBVtx72wo6aCOxDd79qt2NV9W8XfxxefkR8eofTvpqt5/4XXZ/kX5P8RJIXJ3m4qi7uf6Cqbu7upw5tgBUc4cwX3+OXk9ye5JW9PFg9YS67/iucc/Uerj2JDjJzqup7sxWAj3X3nxzhOg/TQWb+xSSvrarbknxfkudV1R919+uPcL2HZ90fSjxXtiTvznd+SHr3DuecSfLlbP0P/+KHTy/b4bx/yun4YPhAMyd5TZIvJDm37lkuM+MV71m2ngVv/8Dws/u53ydtO+DMleSjSd637jmOa+ZLzrk1p+yD4bUv4LmyJbk2yX1JHlu+vmDZ/yNJPrXtvNuy9RsTTyS5c5f3Oi0RONDMSR7P1jPWh5btQ+ueaZc5v2v9Sd6c5M3L60ry/uX455Js7Od+n8Rt1ZmT3JKtxyiPbLuvt617nqO+z9ve49RFwD8bATCY3w4CGEwEAAYTAYDBRABgMBEAGEwEAAYTAYDB/g9I3ziJUe7pHQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# TODO: Plot y3 clamped between 0.0 and 1.0.\n",
    "\n",
    "plt.plot('TODO')\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Subscripts and multiple dimensions\n",
    "----------------------------------\n",
    "\n",
    "Pytorch code is full of multidimensional arrays.  The key to reading this kind of code is stopping to think about the careful, sometimes tangled, use of multiple array subscripts.\n",
    "\n",
    "**Slicing.** As normal in python, you can use `[min:max:stride]` to slice ranges, and multidimensional subscripts like `x[2,0,1,9]` work as you would expect (selecting the 9th entry of the of the 1st of the 0th of the 2nd entry of `x`; and can be used with slices like `x[0:3,2,:,:]`.  The special slice `:` selects the whole range in that dimension.\n",
    "\n",
    "**Unsqueezing to add a dimension, and broadcasting.** While a single integer subscript like `x[0]` eliminates a dimension, the special subscript `x[None]` does the reverse and adds an extra dimension of size one.\n",
    "\n",
    "An extra dimension of size one is more useful than you might imagine, because pytorch (similar to numpy) can combine different-shaped arrays as long as the shape differences appear only on dimensions of size one by **broadcasting** the singleton dimensions.  An example that uses broadcasting to calculate an outer product is illustrated below.\n",
    "\n",
    "**Fancy indexing.** Lots more can be done by passing numerical arrays or boolean array masks as subscripts.  The reshuffling possibilities can get quite intricate; the rules are modeled on the capabilties in numpy.  For details see [Numpy fancy indexing](https://numpy.org/doc/stable/user/basics.indexing.html).\n",
    "\n",
    "Here is a demonstration of simple tensor reshaping."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "m is tensor([[0.1353, 1.2838, 0.2440, 0.5774, 1.3416],\n",
      "        [0.9628, 0.1760, 0.4458, 0.9256, 1.6327]]), and m[1,2] is 0.445751816034317\n",
      "\n",
      "column zero, m[:,0] is tensor([0.1353, 0.9628])\n",
      "row zero m[0,:] is tensor([0.1353, 1.2838, 0.2440, 0.5774, 1.3416])\n",
      "\n",
      "The dot product of rows (m[0,:] * m[1,:]).sum() is 3.1897406578063965\n",
      "\n",
      "The outer product of rows m[0,:][None,:] * m[1,:][:,None] is:\n",
      "tensor([[0.1302, 1.2361, 0.2349, 0.5560, 1.2917],\n",
      "        [0.0238, 0.2259, 0.0429, 0.1016, 0.2361],\n",
      "        [0.0603, 0.5723, 0.1088, 0.2574, 0.5980],\n",
      "        [0.1252, 1.1883, 0.2258, 0.5345, 1.2417],\n",
      "        [0.2208, 2.0961, 0.3983, 0.9428, 2.1904]])\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAgEAAAG6CAYAAACRA5VKAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAA9hAAAPYQGoP6dpAAAwKklEQVR4nO3de5wsdX3n/9fbQwDFc44RvCGQmLjLrohBgY2yEdAEg8YYwSQaNQZzwYWQaEy8EF1E4yUkPo74U7xFVlhiook/xXAzCEq8kQsoEoTgkkUiEiGKnjkqHBA/+0dVc/r0mUvPnJ6p6anX8/Hox0xXV1d9uqen+l3f77eqUlVIkqT+uU/XBUiSpG4YAiRJ6ilDgCRJPWUIkCSppwwBkiT1lCFAkqSeMgRIktRThgBJknrKECBJUk8ZAnoqyXFJaui214SXf+7Qsq+Z5LIlSZNhCNCxwBOAb094uS9vl/uFCS9XkjQhu3RdgDr3har6yqQXWlVfBkgyA0y0lUGSNBm2BEyxJKe2ze2PSfLXSTYnuT3JpiS7JNk/yceSbEnylSQv77pmSdLqYQhYG/4K+CLwLODPgN8D3gKcC1wAHAN8AjgtybELLSzJkW24OHWMeX+0nfesJVcvSeqE3QFrw3uqalP7+yVJngKcBBxbVR8BSHIZ8HTgecCHF1heAfcAPxhj3YN571lC3ZKkDhkC1obzR+5fB/wEcNFgQlV9P8kNwI8stLCq+jvG/GxU1U3jzitJWl3sDlgbbh+5fxfwvaq6c5bpu69MSZKk1c4QIElSTxkCJEnqKUOAdpDkiCTfT3LKyPTBIYlHDk37kXbeM1e6TknSzjEEaDYB1rHj5+P+NEcDfH2WedetTGmSpElJVXVdgzqQ5DjgfcAjgZuq6vtjPOcf23l/aYx570MTIi4F9qyqR+9cxZKkSbMlQDcAdy90AaEkG2gOOzxlvvmGfBi4Gzh858qTJC0XWwJ6KsmewCOGJl01TmvAIpb/48APt3fvqKovTWrZkqTJMARIktRTdgdIktRThgBJknrKECBJUk+t+IVfkgTYG9iy0uuW1oj1NP8/W8pBPZJ2QhdXf9sbuLmD9UprzUZgpusiJE2vLkLAFmguZZcOVr4YX9/8vK5LGMvTNr6/6xIW9O6uCxjTvptf33UJ85qZuZN99309wD7YmtaJoRNtDTyoqr4x9PiPAW8Gnkyzjb0ceEVVfX4n1nkVzXk6AC6oqqe3008FXtNO/25V3X/oOZcBRwB/W1VHjyzvR4EbgZdV1ZuXWlcXkhwEfGFo0i9V1Yc6KmfqdXYd+LD6Q8CGDbt2XcJYOvsjLsL6rgsY04YNU3OlZbsCuncs8O/AtwcTkjwI+DTwLeDXgTuBk4HLkhxaVdcvcV2/CuwBfGSOx58A3DPHYz+b5MlV9Yklrnu1+TLN630ccEbHtUy9afj+kKTV6AtV9ZWRaS8DHgQcVlU3AST5DPCvwOuAZy9lRVX1z+2yts7x+N/P8dQv02zn/6QNIVMfHKvqe8DfJ5maxL6aeXSApN4ZuiLmY5L8dZLNSW5PsinJLkn2T/KxJFuSfCXJy8dc9DHAJwYBAKCqZmhOo/3zSVZ6x+tu4FXAwYwRQJI8OslHk3wryZ1JrkryayPzHNm+d7+S5A1Jbkkyk+SSJPvPssyfSXJpO8/3knw2yU9P7BVqpxgCJPXZXwFfBJ4F/Bnwe8BbgHOBC2i/1IHTkhw734KS3Bf4ceDqWR6+Grgv8GMLLGPwBXvqol7F/D4IXAm8PskPzbPu/YHPAQcAv0vT3XEtcNYcIeiNwI8AvwkcD/wn4Lwk915RNMnzgYtpBrD+GvDLwO3A344TBJKc1b4fPzrG69QS2B0gqc/eU1Wb2t8vSfIU4CTg2Kr6CNw7wO7pwPNo9ujn8sM0Q51un+WxwbQ9F6inaPr2fzBW9WOoqkryCuAS4EXA2+eY9VRgV+BJVfXVdtqFSR4AvCbJu6tq89D811bV8wd3ktxDE6oOpWmuvx/wVuD8qjpmaL4Lgc/ThIifXKD8e9rb1HdjrFa2BEjqs/NH7l9H84Vz0WBCe2GtG2j2escx3xfWvF9mVfV3VbVLVb1uzHWNV1DVpTR75KckmWuc7pOBS4cCwMBZwP1oBuMN+5uR+4MWkMH7dBjwQODstotll7Y75D7Ax4BDk+yxQN2/0b4fN803n5bOECCpz0b32u8CvldVd84yfaGBaN+i+ZKfbW//gXOsbyW9AtgL+IM5Ht+T5miHUbcMPT7smyP3B4MW79v+fEj780M0YxOGb6+gaTV5IOqU3QGSNAFVdUeSG4ADZ3n4QOAO4P+ubFXbVNVVSf4SeClw4SyzfBN42CzT925/fmOWx+YzmP93gLmOXrh1kcvUhBkCJGlyPgK8JMm+g2b1tvn9WOBv2q6FLr0a+EW2nWBo2KXAMUn2rqpbhqa/APgec3+Rz+WzNOdQeFRVzTUOQR2zO0CSJufNNHvUFyR5ZpKn0ow72J1m4N29Zhv5nuSIJN9PcspyFFdVNwLvBJ46y8OvpWmq/2SS5yV5apI/B34OOHVkUOA46/oOTSvAi5J8IMkvJjk8ybOSvC7JO4fnb9+Ly0amndm+H+OOx9AiGQIkaUKq6j+AJ9KcHOhstvWHH1lV/zIy+/1pugi+PTQtwDqWd9v8ema55kR7NsPDgOtpzsR3LvBo4IVV9adLWVFV/TnwJJrX+m6aIxTeSnO2v0sH8yUZnO54dEzCuva22k8wO7XsDpDUO1V1KiN75u3044DjZpl+5CyLWZdkl9Em/qr6V5rzCyzkp4AzqurbQ8+9jFm+8Npj7+f8ImxH3VdV3Xvq4Dlqpr3OwcY5HrsGeMZ8Rc9VY3v2xNmmfwr41HzLBA6nGVT5xpHnHscsf4/29a4bna7FsyVAkpbmBuDuJHst9olJDqA57O60MZ9yJU2LwlzN4ncDi2quX2WeBHxgcHrk+bQXELqbplVBO8mWAElanPNoTogz8O3FLqCqvgRsWMRTnksTGkbX9x62netgrgsIrXpV9bJFzH4927///zrhcnrFECBJi1BV32THY+SXe53XzjH9FrYdx98LVXUHcEXXdawVS+oOSHJikhvbC0xcmeSJky5MkiQtr0WHgCTPBk4H3gA8luba2Rcl2W+ypUmSpOW0lJaAlwJnVtV7q+q6qnoJ8FXghIlWJkmSltWixgQk2ZXmutR/PPLQxTTHl872nN2A3YYmzXXxCklTIkloTie7petapCm2Hrilqjq7SuJiBwbuRXNs5uj5nm8FHjrHc05m9lNUSppeewM3d12EtAbsA3ytq5Uv9eiA0dSSWaYNvAnYNHR/PW48pGm3BZpz4a72U7l9ffPzui5hLE/b+P6uS1jQu7suYEz7bn591yUsaGbmTvbd9/XQcWvaYkPAN2iORR3d638wc1wNqqq2su0SkzStiJLWgrD6Q8CGDbt2XcJYpuF47Wnpy92wYaGrPmtgUQMDq+oumjNXHTXy0FHA5yZVlCRJWn5LCZ+bgHOSXAFcDhwP7Ae8a5KFSZKk5bXoEFBVH0yyJ3AK8DDgGuBpVXXTpIuTJEnLZ0ndUFX1DuAdE65FkiStIK8iKElSTxkCJEnqKUOAJEk9ZQiQJKmnDAGSJPWUIUCSpJ4yBEiS1FOGAEmSesoQIElSTxkCJEnqKUOA1FNJTkxyY5I7k1yZ5Ild1yRpZRkCpB5K8mzgdOANwGOBTwMXJdmvy7okrSxDgNRPLwXOrKr3VtV1VfUS4KvACbPNnGS3JBsGN2D9CtYqaZks6SqCk/D1d8OG+3a19vHsm/d1XcJY3th1AWPY7/SuKxjX73ddwAJmgFfv1BKS7AocDPzxyEMXA4fN8bSTgdfs1IolrTq2BEj9sxewDrh1ZPqtwEPneM6bgI1Dt32WrTpJK6azlgBJnauR+5llWjNj1VZg670zJstYlqSVYkuA1D/fAO5hx73+B7Nj64CkNcwQIPVMVd0FXAkcNfLQUcDnVr4iSV2xO0Dqp03AOUmuAC4Hjgf2A97VaVWSVpQhQOqhqvpgkj2BU4CHAdcAT6uqm7qtTNJKMgRIPVVV7wDe0XUdkrrjmABJknrKECBJUk8ZAiRJ6ilDgCRJPWUIkCSppwwBkiT1lCFAkqSeMgRIktRThgBJknrKECBJUk8tOgQkOTzJeUluSVJJnrkMdUmSpGW2lJaAPYAvAidNuBZJkrSCFn0Boaq6CLgIIMmC8yfZDdhtaNL6xa5T0upUXRcwhpmZu7ouYSzf77qAMWzpuoAxzczc2XUJC1otNa7EVQRPBl6zAuuRtHLWA6yOzdj8Nm58f9clrBmP7rqAcW18ddcVLMZ6YKarla9ECHgTsGno/nrg5hVYr6TlcwuwD5PdORxsGya93EmzzsmZhhph+epcT/O/1JllDwFVtRXYOrg/TheCpNWtqgr42iSXObRt2FJVne0ZLcQ6J2caaoRlrbPz1+whgpIk9ZQhQJKknlp0d0CS+wOPHJr0iCQHAbdX1b9NqjBJvbMVeC1D3YerlHVOzjTUCNNT56ItZUzAIcAnh+4PBv2dDRy3swVJ6qd2/NCpXdexEOucnGmoEaanzqVYynkCLgMc3SdJ0pRzTIAkST1lCJAkqacMAZIk9ZQhQJKknjIESOpckhOT3JjkziRXJnli1zWNmobLqCc5Ock/JdmS5LYk5ybZv+u6RiU5IcnVSWba2+VJntp1XfNp39tKcnrXtUySIUBSp5I8GzgdeAPwWODTwEVJ9uuyrllMw2XUjwDOAB4PHEVzBNjFSfbotKod3Qy8kuaQ80OATwAfTXJAp1XNIcmhwPHA1V3XMmkrcQEhSZrPS4Ezq+q97f2XJPlZ4ASaq5CuCou9jHoXquro4ftJXgjcBhwMfKqTomZRVeeNTHpVkhNowsuXOihpTu0J8t4P/BYwVZcnHIctAZI6k2RXmi+oi0ceuhg4bOUrWnM2tj9v77SKeSRZl+Q5NC0tl3ddzyzOAC6oqku6LmQ52BIgqUt7AeuAW0em3wo8dOXLWTvSNFdsAj5TVdd0Xc+oJAfSfOnvDnwHOKaqru22qu214eRxwKFd17JcDAGSVoMauZ9Zpmlx3g48BviprguZw/XAQcADgGcBZyc5YrUEgST7Am8FnlJVd3Zdz3IxBEjq0jeAe9hxr//B7Ng6oDEleRvwDODwqrq563pmU1V3ATe0d69oB9+9GHhRd1Vt52Caz+GVQ2NA1gGHJzkJ2K2q7umquElxTICkzrRfBFfSjGQfdhTwuZWvaLql8XbgWODJVXVj1zUtQoDdui5iyKXAgTStFYPbFTSDBA9aCwEAOmwJmLmjqzWP7wddFzCmKXgrmZmWxrSZma4rmNfMtvrWJ9lSVWuhyXwTcE6SK2j6iI8H9gPe1WlVI6bkMupnAM8FfgHYkmTQwrK5qlbNpiLJG2mOtPgqsB54DnAkcPQ8T1tRVbUF2G4sRZLvAt9cjWMsliorvQ1J8nCaY0Ql7ZyNVbW6U8uYkpwIvBx4GM2G9/eqatUc0gaQ5Ei2v4z6wNlVddyKFjOHJHNt0F9YVWetZC3zSXIm8NM0f+/NNMffn1ZVH++0sAUkuQy4qqpe0nEpE9NFCAiwN7BlQotcTxMq9pngMpeDdU5Wn+tc3y5rrbQESOrIincHtButr01qeUMDNras5r0i65ysnte5al+vpOniwEBJknrKECBJUk+thRCwFXht+3M1s87Jsk5J2kkrPjBQkiStDmuhJUCSJC2BIUCSpJ4yBEiS1FOGAEmSesoQIElST019CEhyYpIbk9yZ5MokT+y6pmFJDk9yXpJbklSSZ3Zd02ySnJzkn5JsSXJbknOT7N91XcOSnJDk6iQz7e3yJE/tuq6FtO9tJTm961okadhUh4AkzwZOB94APBb4NHBRkv26rGvEHsAXgZO6LmQBR9BcgezxNJdx3QW4OMkenVa1vZuBVwKHtLdPAB9NckCnVc2jvUb68TQXSJGkVWWqzxOQ5B+Az1fVCUPTrgPOraqTu6tsdu0Vvo6pqnO7rmUhSR4E3AYcsdqu5jYsye3Ay6rqzK5rGdVeevbzwInAq1ljVx+TNP2mtiUgya7AwcDFIw9dDBy28hWtORvbn7d3WsUckqxL8hyalpbLu65nDmcAF1TVJV0XIkmzWfGrCE7QXsA64NaR6bcCD135ctaO9nLPm4DPVNU1XdczLMmBNF/6uwPfoWlZubbbqnbUBpTHAYd2XYskzWWaQ8DAaH9GZpmmxXk78Bjgp7ouZBbXAwcBDwCeBZyd5IjVFASS7Au8FXhKVd3ZdT2SNJdpDgHfAO5hx73+B7Nj64DGlORtwDOAw6vq5q7rGVVVdwE3tHevaAfevRh4UXdV7eBgms/hlU2jCtC0Wh2e5CRgt6q6p6viJGlgascEtF8GV9KMZB92FPC5la9ouqXxduBY4MlVdWPXNY0pwG5dFzHiUuBAmhaLwe0K4P3AQQYASavFNLcEQNNvfU6SK2j6iY8H9gPe1WlVQ9oR4o8cmvSIJAcBt1fVv3VT1azOAJ4L/AKwJcmghWVzVd3RXVnbJHkjcBHwVWA98BzgSODoDsvaQVVtAbYbS5Hku8A3V9sYC0n9NtUhoKo+mGRP4BTgYTQb3qdV1U3dVradQ4BPDt3f1P48GzhuxauZ2+Awy8tGpr8QOGtFK5nbQ4BzaP7Wm2mOvT+6qj7eaVWSNKWm+jwBkiRp6aZ2TIAkSdo5hgBJknrKECBJUk8ZAiRJ6ilDgCRJPWUIkCSppwwBkiT1lCFAkqSeMgRIktRThgBJknrKECBJUk8ZAiRJ6ilDgCRJPWUIkCSppwwBkiT1lCFAkqSeMgRIktRThgBJknrKECBJUk8ZAiRJ6ilDgCRJPWUIkCSppwwBkiT1lCFAkqSeMgRIktRThgBJknrKECBJUk8ZAiRJ6ilDgCRJPWUIkCSppwwBkiT1lCFAkqSeMgRIktRThgBJknrKELBCkhyXpIZuew09dkCSdyS5PMl328ePnMA6rxpa3/lD008dmv6dnVj+We0yvpRk3SyPV5K3L3X5k5TkoJH3/xe7rkmSumYIWHnHAk8Avj007RDgmcDtwKUTXNevtuv6+hyPPwF40gTW8yjguAksZzl9meb1/nbXhUjSarFL1wX00Beq6isj086pqrMB2j3Un5/Eiqrqn9tlbp3j8b+fwGq+C3weeG2Sv6iqOyawzImrqu8Bf59k965rkaTVwpaARRhqRn9Mkr9OsjnJ7Uk2Jdklyf5JPpZkS5KvJHn5OMutqh8sd+3L7BXAw4EXLzRjkv2S/HmS25JsTXJdkt9Pcp+heX60fZ//IMlLk9yY5Dttd8njZ1nmIUn+pv1b3JnkC0l+ebIvUZLWHkPA0vwV8EXgWcCfAb8HvAU4F7gAOAb4BHBakmOXu5gkR7ZfmqdOYFmDsQvHjfucqroc+AjwiiQPnGfZDwI+BzwF+J/AM4BLgDcDs40d+G3gKOAlwPOAPYALk2wcWuaTgM8CDwD+B/ALwFXABxfzGiSpjwwBS/Oeqnp9VV1SVa+g+dI5CfjDqnpbVV0CHA/8B82X13Ir4B5gEi0KP1jisk4G1gN/OM88L6VpMXh6Vb27qv62qn4HeCfwP5L855H5t7TzfrSqPgr8BvDDwFOH5nkH8CXgyVX1V+0yXwicD7xxuIVBizffgNb28R9L8uEk325baz6e5HE7uc5FD2hNclk7/WOzLO/elqWdqasLSx3Q2u4YnLXM5Q2v77LZ/mbtYy9I8oEk1yf5QZKvTGB9c74vbSvsYPqSByYPLeOVszw2+L84ZKnLn6Qkpy91sLcbyKU5f+T+dTRfxBcNJlTV94EbgB9Z7mKq6u+qapeqet0ElvW/22X970U+73rgTOCkJPvNMduTgWur6h9Hpp8FpH182AVVdc/Q/avbnz8CkOSRwH8B3t/e32VwAy4EHgbsv5jXoTntMKC1bdn5NPCfgV8HfhnYHbgsyc687zszoPVnk4x+jqbZ2ANakzwyye8m+aGR6U9I8mvLVeCQL9DU+vsj038VOAD4R+BfJ7Suhd6XC9vH3zyBdb1yvhbOVeItNK/3wsU+0YGBS3P7yP27gO9V1Z2zTN+wMiWtCqcCzwf+CJhto7Mn8JVZpt8y9Piwbw7fqaqtSQDu2056SPvzzcz9z77XHNO1OLMNaH0Z8CDgsKq6CSDJZ2g29K8Dnr2UFe3EgNYv02zT/iTJoVVVS1n/arLIAa23Aw8FLqc5ymjvJB8A7kfzv7ncZub42/zsYNxT20rw6J1d0Rjvy39MaODzJcCRwKvYMdysGu3/301J/mOxz7UlQBNTVf8OnA48P8ljZpnlmzR756P2bn9+Y5GrHMz/JuDQOW5XLXKZa0qWaTBr6xjgE4MAAFBVM8CHgZ9vW2RW0t00G+uDGSOAJHl0ko8m+VY7oPSq0T3mofE2v5LkDUluSTKT5JLZWjuS/EySS9t5vpfks0l+emKvcB5VdXtV/SHwHOC5wM8An6yqZ1TV5+d7bvu3Pz/J09MMrL0jzaDdp7ePH9fe/26Sf1xMM/iUD3wetHD+dpIFW3WTPCPNAObvtf9TH0/yhJF5Bv+TByT5y/Z/8tYk/ytD453aeZPkxPazeUf7Wf1Qkh+b1As0BGjSTqPZI/njWR67FHhUduwzfgFNd8onF7Oitgvi/wA/UVVXzHHbsoTXsBZNdDBrkvsCP862LpphV9O01sy7ocoEB7QO+SBwJfD60WbxkXXvTzNI9QDgd2m6O64FzpojCL2RphvqN2nG+/wn4LwMnSQryfOBi4EZmpawX6b5X/jbcYJAtp1860fHeJ2zPf8BSV4H/CXwFzR7sU9ug8444zR+giZQn0bzfmwGPpzktTSv+w9pxjhtBM5vPwPLbmffl5FlXZZksS1Ep9KMk/qjBZb9XOCjNH//X2HbGKbLkvzULE/5/2lar55Fs718Ls3/5LB30+xYXUJzLpkTaT6zn0vyECbA7oBVIMn9gKe1dweHwB2RZhDWd6vqoqF5z6LZwDxi0Dyb5AiaL9jX7ey4gCQvAP4X8OvD4wJmW+9sqmomyRvY8cNMO+0FwAVJTgFuAn6O5oP9zqr68hJKfhFwUZK/pRlb8DXggcB/BR5XVb+0hGWuRe+pqk3t75ckeQrNYNZjq+oj0GwggafTbOg/vMDyfphmHMdo1xhD00a7d0ZNckBrs8CqSvIKmo3mi5j9qBNoNuy7Ak+qqq+20y5M8gDgNUneXVWbh+a/tqqeP7iT5B6aYHUoTbP0/YC3AudX1TFD811Icx6NNwI/uUD597S3pXZjPIimdeww4L8DD6mq49o90QPbOuazJ/D4qvpaW/stNC1pvwU8sm2Cp/0SPZempeG8Jda6GDv7vsy2rLFV1deTvAU4Ocmbq2qH4JtmAPKfAv8MPHWo++NCmu6x02j+JsPOrKo/bX+/JM0Yp19P8hvt5/jxNO/97w/975Lk0zTh4aU0h2fvFFsCVocHA3/d3gb9Tqe29985Mu/9gTvY/oyDAdYxmb/nfeZY1mzrncs7gBtHJ1bVf9BsoD5Bs8dxPvCzwMuB31lKsVX1SeC/tXWdTrPxfyfNBuqSpSxzjVquwazzbZjn3WhPckDryHIvpdkjPyXJ+jlmezJw6VAAGDiLpg/9CSPT/2bk/naDVGk+1w8Ezs72A1TvA3wMODTJHgvU/Rvt+3HTfPPN8/z/U1X/X1XdPTL98sHJyBZw1SAAtK5rf142CAAj05d90DPs/Psysqyfrqql7Pz+CU24PW2Ox/en6dY8Z7j7o6q+Q7PH//g2KA6b7TO1O833ATSBvIA/H/lMfZ2mVe/IJbyOHdgSsAhVdSqzDLCpquOY5bS5VXXkLItZl2SXdoM7mO8rNF/k4/gp4Iyq+vbQ8y+b7fltU+Wcy20/UDU8Ar+qzqLZEI6z3uOY/XXfxRxNwVX1byxw2OR870dV7TC9Tebj9AHvQhNw+mjSg1m/RbOBmm1vfzCSerZWgpXyCpo93z8A3jfL43sC/z7L9LEGqQKDQYujg1Q/NE9ND6Q5w+aya7cJly3yadv9varqrjQDcWf77EDzhdULbQvn64HT05ybZNTg8zLXZ+o+NK1nw2FqnM9UgFvnKOv/LlT3OAwBK+8GaA6vqqpFDYRLcgDNXspcaXTUlTT9fADXzPL43TQbpftPeL2rTpKDaA5h0gRU1R1JbqBpZh51IE2r0UQ2UktRVVcl+UuaJtPZDptarkGqvwPMNSp9ro25psM7ac6Keho7ttAOvtDn+kz9gCY4L8Y3aIL2E9kWEIbNevTMYhkCVs55NP2HA99e7AKq6kss7pDD59J8eY+u7z1sax5esH9sCetdja5n+/d/Uscr99lHgJck2XfQrN42vx8L/M1wa1dHXg38IvCaWR67FDgmyd5VdcvQ9BfQ7K0t9vCyz9L8jz2qqlbFlTM1WW3LyKtpzksyGhKvpxmP9Nx23EABtF1AzwIuH+lSGcf5wCuBh1fVX+1c9XMzBKyQqvomOzb/LPc6r51j+i1sa/bshWoubHRF13WsMW+mORHMYKDnVpqN1u6MdJst94DW2VTVjUkGe2+jXkvT5/rJdkT97TTdVD8HvHxkUOA46/pOkt+hGRPwQJpugdtoBuv9BPCgqjphMH87uO7vhrsMk5xJ8x79+CT6v1eLJI+iudIoNOcxuF+2neHv2uHt1HK/L0kuBY4YHRcw23rn8Jc0XUzDZy2lqn7QHlXyfpojJ94N7EZzLo0H0PxfLEpVfTbJe4D3pTkk81M0LbcPo+me/eeqGm2RWDQHBkpaknag5xNpWlXOpvniuxs4sqr+ZWT25R7QOpfX0xyytZ328NLDaPbgzqAZ7f5o4IVDI7YXpar+nOZMhvenObTrEpojBh7H0CXCkwy630b7j9e1t3HHB02LX2bbwOeDaYLR4P69F/paofdlsKx7zbPeHbR7+LOOyK+qv6A5jG9PmkNV30fz2XtSVX1mKcVW1YtojuI5HPgAzeG8r6O5jsromVeXJDX9J9WStEzSXITpfcAjgZuW2sSf5Os0I6dfNsa8gw3+DcA1VTU4Yc2pNE37P8TIgNZpkuRpNE29P1Ht2REXmH8X4AiaUPFLVTXf4MNOtIeXBvhp4Ae1hBMETep9SXNtgr+jOU5/wVoWu97VqD1E8T40JzZ6VlXNO85rmN0BUyrNsN29aS6yMy3WA7eUyXMaTd2A1lXsScAHxvyiO4jpGdB6OM3f5gKarpbFmuT78oL2dgbNnvRE1ruKbWJbt9eijkCxJWBKJXk4cHPXdSzBPiPHImsVS7In8IihSVct94C/tg/53gGtVTUIIHuzbfT+PVU1LV+OS9aele+AoUn/WlWLHWW+7NozMA7OyXDv32wZ1zfn+5LkQJr+eIDb2sOS17Qk+7LtMNVF/W8YAqZUkg3A5q+eBhum4GjdmTth36YnbWM155eXJHXM7oApt2F32LAiZ/CWJK01Hh0gSVJP2RIgadGmdGCqtNp0PljaECBpKfZmOgemSqvNPjRnG+yEIUDSUmyB5iw4q/2KTJdsPrrrEsby0o0f67qEBS3pLEodWLf5xK5LWNDMzF3su+97oePWNEOApCVbx+rfiGzY8ENdlzCWXbsuYAzTcgGRdRt2W3gmAQ4MlCSptwwBkiT1lCFAkqSeMgRIktRThgBJknrKECBJUk8ZAiRJ6ilDgCRJPWUIkCSppwwBkiT1lCGgQ0lOTHJjkjuTXJnkiV3XJEnqD0NAR5I8GzgdeAPwWODTwEVJ9uuyLklSfxgCuvNS4Myqem9VXVdVLwG+CpzQbVmSpL4wBHQgya7AwcDFIw9dDBw2x3N2S7JhcAPWL3OZkqQ1zhDQjb1orsJ668j0W4GHzvGck4HNQ7ebl6069YJjUiQZArpVI/czy7SBNwEbh277LGNdWuMckyIJDAFd+QZwDzvu9T+YHVsHAKiqrVU1M7gBW5a5Rq1tjkmRZAjoQlXdBVwJHDXy0FHA51a+IvWJY1IkDRgCurMJ+M0kv57kvyZ5C7Af8K6O69La55gUSQDs0nUBfVVVH0yyJ3AK8DDgGuBpVXVTt5WpRxY7JmXT0P31GASkqWcI6FBVvQN4R9d1qHeWNCYF2Dq4n2TZipO0cuwOkHrGMSmSBmwJkPppE3BOkiuAy4HjcUyK1DuGAKmHHJMiCQwBUm85JkWSYwIkSeopQ4AkST1lCJAkqacMAZIk9ZQhQJKknjIESJLUU4YASZJ6yhAgSVJPGQIkSeopQ4AkST1lCJAkqae8dsCUe+iLYRqu7F5dFyBJ2oEtAZIk9ZQhQJKknkqVDbXTKMkGYPN9mZ7ugDuaXzdW1UynxWinTdPn7/ldFzCmD3VdwBj27rqAMT2x6wLGcBdwZvNrp9tEWwIkSeopQ4AkST1lCJAkqacMAZIk9ZQhQJKknjIESJLUU4YASZJ6yhAgSVJPGQIkSeopQ4AkST1lCJAkqacMAZIk9ZQhQJKknjIESJLUU4aAjiQ5PMl5SW5JUkme2XVNkqR+MQR0Zw/gi8BJXRciSeonQ0BHquqiqnp1VX2461rUP7ZESQJDwNRIsluSDYMbsL7rmjTVbImSxC5dF6CxnQy8pusitDZU1UXARQBJOq5GUldsCZgebwI2Dt326bYc9YktUdLaZAiYElW1tapmBjdgS9c1qVdOBjYP3W7uthxJk2AIkDQOW6KkNcgxAR1Jcn/gkUOTHpHkIOD2qvq3bqqSZldVW4Gtg/uOI5DWBkNAdw4BPjl0f1P782zguBWvRpLUO4aAjlTVZYC7U+qELVGSwBAg9ZUtUZIMAVIf2RIlCTw6QJKk3jIESJLUU4YASZJ6yhAgSVJPGQIkSeopQ4AkST1lCJAkqacMAZIk9ZQhQJKknjIESJLUU4YASZJ6yhAgSVJPGQIkSeopryI45V4L3LfrIsZwB/DyrouQJG3HECBpyTay+psT3/1DXVcwnn+4u+sKFvYzXRcwpjdv7LqChc0UnDnTdRWr//9XkiQtE0OAJEk9ZQiQJKmnDAGSJPWUIUCSpJ4yBEiS1FOGAEmSesoQIElSTxkCJEnqKUOAJEk9ZQiQJKmnDAGSJPWUIUCSpJ4yBEiS1FOGAEmSesoQIElSTxkCOpLk5CT/lGRLktuSnJtk/67r0trnZ0/SgCGgO0cAZwCPB44CdgEuTrJHp1WpD/zsSQKaf351oKqOHr6f5IXAbcDBwKc6KUq94GdP0oAhYPXY2P68fbYHk+wG7DY0af2yV6S+mPezB37+pLXK7oBVIEmATcBnquqaOWY7Gdg8dLt5hcrTGjbmZw/8/ElrkiFgdXg78BjgV+aZ5000e2yD2z4rUJfWvnE+e+DnT1qT7A7oWJK3Ac8ADq+qOfeuqmorsHXoeStQndaycT974OdPWqsMAR1pm2HfBhwDHFlVN3ZcknrCz56kAUNAd84Angv8ArAlyUPb6Zur6o7uylIP+NmTBDgmoEsn0PStXgb8+9Dt2R3WpH7wsycJsCWgM1Vlp6o64WdP0oAtAZIk9ZQhQJKknjIESJLUU4YASZJ6yhAgSVJPGQIkSeopQ4AkST1lCJAkqacMAZIk9ZQhQJKknjIESJLUU4YASZJ6yhAgSVJPeRVBSUv2cKZgI3LXo7quYCw/mWu7LmFBp3RdwLi+fUzXFSxs5m7YeH7XVaz+/1/N70X/Ezbs3nUVC5u5E17+R11XIUkaZneAJEk9ZQiQJKmnDAGSJPWUIUCSpJ4yBEiS1FOGAEmSesoQIElSTxkCJEnqKUOAJEk9ZQiQJKmnDAGSJPWUIUCSpJ4yBEiS1FOGAEmSesoQIElSTxkCJEnqKUOA1ENJTkhydZKZ9nZ5kqd2XZeklWUIkPrpZuCVwCHt7RPAR5Mc0GlVklaUIaAj7ompS1V1XlVdWFVfbm+vAr4DPL7r2iStnF26LqDHBntiN7T3f41mT+yxVfWl7spS3yRZB/wSsAdw+Rzz7AbsNjRp/QqUJmmZGQI6UlXnjUx6VZITaPbEDAFadkkOpPnS352mFeCYqrp2jtlPBl6zUrVJWhl2B6wCSdYleQ4L7Ikl2TC44Z6Ydt71wEE0wfOdwNlJHjXHvG8CNg7d9lmJAiUtL1sCOuSemLpUVXexrTvqiiSHAi8GXjTLvFuBrYP7SVakRknLy5aAbrknptUkbN/vL2mNsyWgQ+6JqStJ3ghcBHyVpmvpOcCRwNEdliVphRkCVhf3xLRSHgKcAzwM2AxcDRxdVR/vtCpJK8oQ0BH3xNSlqvqNrmuQ1D1DQHfcE5MkdcoQ0BH3xCRJXfPoAEmSesoQIElSTxkCJEnqKUOAJEk9ZQiQJKmnDAGSJPWUIUCSpJ4yBEiS1FOGAEmSesoQIElSTxkCJEnqKa8dIGnJHgXs2nURC/pS1wWM5emk6xIWtOFDXVcwrg93XcAYZoCNXRdhS4AkSX1lCJAkqafsDph2f7AZNmzouoqFzczAH3Xf9CVJ2saWAEmSesoQIElSTxkCJEnqKUOAJEk9ZQiQJKmnDAGSJPWUIUCSpJ4yBEiS1FOGAEmSesoQIElSTxkCJEnqKUOAJEk9ZQiQJKmnDAGSJPWUIUDquSQnJ6kkp3ddi6SVZQiQeizJocDxwNVd1yJp5RkCpJ5Kcn/g/cBvAd9aYN7dkmwY3ID1K1GjpOVlCJD66wzggqq6ZIx5TwY2D91uXs7CJK0MQ8AqYJ+sVlqS5wCPo/lyH8ebgI1Dt32WqTRJK2iXrgvoO/tktdKS7Au8FXhKVd05znOqaiuwdWgZy1SdpJVkS0CH7JNVRw4GHgxcmeT7Sb4PHAH8bnt/XbflSVophoBu2SerLlwKHAgcNHS7giaQHlRV93RVmKSVZXdAR4b6ZA8d8ylvAjYN3V+PQUBLUFVbgGuGpyX5LvDNqrpm9mdJWosMAR2wT1aStBoYArox3Cc7mLYOODzJScBuNslqJVXVkV3XIGnlGQK6MeiTHfY+4F+A0wwAkqSVYAjogH2ykqTVwKMDJEnqKVsCVgn7ZCVJK82WAEmSesoQIElSTxkCJEnqKUOAJEk9ZQiQJKmnDAGSJPWUIUCSpJ4yBEiS1FOeLEjSkt3VdQFjmJmZ6bqEsXyv6wLGMDMNRQJMwd98tXwuU1Vd16AlSLIB2Lx582Y2bNjQdTkLmpmZYePGjQAbq2p1fPq1ZEkeDtzcdR3SGrBPVX2tq5XbEiBpKW4B9gG2THCZ62mCxaSXO2nWOTnTUCMsX53raf6XOmMIkLRo1TQhTnTvJcng1y2rubXIOidnGmqEZa2z89fswEBJknrKloApt1oGlyxkWuqUpD4xBEyv9QD77rtv13Us1npWQROYVqWtwGvbn6uZdU7ONNQI01Pnonl0wJRK00m1N5MfTLOcA3XWA7eUHzpJWhUMAdrO4NBDPJRPktY8BwZKktRThgBJknrKEKBRa3YAjCRpe44JkCSpp2wJkNS5JCcmuTHJnUmuTPLErmsaleTwJOcluSVJJXlm1zWNSnJykn9KsiXJbUnOTbJ/13WNSnJCkquTzLS3y5M8teu65tO+t5Xk9K5rmSRDgKROJXk2cDrwBuCxwKeBi5Ls12Vds9gD+CJwUteFzOMI4Azg8cBRNOeCuTjJHp1WtaObgVcCh7S3TwAfTXJAp1XNIcmhwPHA1V3XMml2B0jqVJJ/AD5fVScMTbsOOLeqTu6usrklKeCYqjq361rmk+RBwG3AEVX1qa7rmU+S24GXVdWZXdcyLMn9gc8DJwKvBq6qqpd0WtQE2RIgqTNJdgUOBi4eeehi4LCVr2jN2dj+vL3TKuaRZF2S59C0tFzedT2zOAO4oKou6bqQ5eBpgyV1aS9gHXDryPRbgYeufDlrR3tW0U3AZ6rqmq7rGZXkQJov/d2B79C0rFzbbVXba8PJ44BDu65ludgSoHtNw+AsrVmj/ZKZZZoW5+3AY4Bf6bqQOVwPHEQzfuGdwNlJHtVpRUOS7Au8FXh+Vd3ZdT3LxRAgYKoGZ2lt+QZwDzvu9T+YHVsHNKYkbwOeATypqm7uup7ZVNVdVXVDVV3Rjv34IvDirusacjDN5/DKJN9P8n2agZe/295f1215k2EI0MBLgTOr6r1VdV078OWrwAnzP01auqq6C7iSZiT7sKOAz618RdMtjbcDxwJPrqobu65pEQLs1nURQy4FDqRprRjcrgDeDxxUVfd0VdgkOSZAw4Oz/njkIQdnaSVsAs5JcgVNH/HxwH7AuzqtakQ7SvyRQ5MekeQg4Paq+rduqtrBGcBzgV8AtiQZtLBsrqo7uitre0neCFxEs6OxHngOcCRwdIdlbaeqtgDbjaVI8l3gm6txjMVSGQIEDs5Sh6rqg0n2BE4BHkaz4X1aVd3UbWU7OAT45ND9Te3Ps4HjVrya2Q1a7i4bmf5C4KwVrWR+DwHOofl7b6Y5/v7oqvp4p1X1kOcJEEn2Br4GHFZVlw9NfxXwq1X1XzorTpK0bBwTIHBwliT1kiFADs6SpJ5yTIAGpmJwliRpcgwBAqZqcJYkaUIcGChJUk85JkCSpJ4yBEiS1FOGAEmSesoQIElSTxkCJEnqKUOAJEk9ZQiQJKmnDAGSJPWUIUCSpJ4yBEiS1FOGAEmSeur/AQI9gGEAR5unAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 500x500 with 4 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import torch\n",
    "from matplotlib import pyplot as plt\n",
    "\n",
    "# Make an array of normally distributed randoms.\n",
    "m = torch.randn(2, 5).abs()\n",
    "print(f'm is {m}, and m[1,2] is {m[1,2]}\\n')\n",
    "print(f'column zero, m[:,0] is {m[:,0]}')\n",
    "print(f'row zero m[0,:] is {m[0,:]}\\n')\n",
    "dot_product = (m[0,:] * m[1,:]).sum()\n",
    "print(f'The dot product of rows (m[0,:] * m[1,:]).sum() is {dot_product}\\n')\n",
    "outer_product = m[0,:][None,:] * m[1,:][:,None]\n",
    "print(f'The outer product of rows m[0,:][None,:] * m[1,:][:,None] is:\\n{outer_product}')\n",
    "\n",
    "fig, ((ax1, ax2), (ax3, ax4)) = plt.subplots(2, 2, figsize=(5, 5), dpi=100)\n",
    "def color_mat(ax, m, title):\n",
    "    ax.set_title(title)\n",
    "    ax.imshow(m, cmap='hot', vmax=1.5, interpolation='nearest')\n",
    "    ax.get_xaxis().set_ticks(range(m.shape[1]))\n",
    "    ax.get_yaxis().set_ticks(range(m.shape[0]))\n",
    "color_mat(ax1, m, 'm[:,:]')\n",
    "color_mat(ax2, m[0,:][None,:], 'm[0,:][None,:]')\n",
    "color_mat(ax3, m[1,:][:,None], 'm[1,:][:,None]')\n",
    "color_mat(ax4, outer_product, 'm[0,:][None,:] * m[1,:][:,None]')\n",
    "fig.tight_layout()\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise\n",
    "\n",
    "Use `torch.mm` to compute `outer_product` and `dot_product`.\n",
    "\n",
    "Explain to yourself why order matters when using torch.mm but not when using `*`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "False\n",
      "False\n"
     ]
    }
   ],
   "source": [
    "# TODO Use torch.mm to compute outer_product and dot_product.\n",
    "\n",
    "outer = 'TODO'\n",
    "print(outer == outer_product)\n",
    "dot = 'TODO'\n",
    "print(dot == dot_product)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Devices and types\n",
    "-----------------\n",
    "\n",
    "One of the big reasons to use pytorch instead of numpy is that pytorch can do computations on the GPU.  But because moving data on and off of a GPU device is more expensive than keeping it within the device, pytorch treats a Tensor's **computing device** as pseudo-type that requires explicit declaration and explicit conversion.  Here are some things to know about pytorch devices and types:\n",
    "\n",
    "**Single precision CPU default.** By default a torch tensor will be stored on the CPU and will store single-precision 32-bit `torch.float` values.\n",
    "\n",
    "**Specifying data type.** To store a different data type such as integers, use the argument `dtype=torch.long` when you create the Tensor.  For example, `z = torch.zeros(10, dtype=torch.long)`.  This is similar to numpy with minor differences.  See the [Tensor reference](https://pytorch.org/docs/stable/tensors.html) for all the types.\n",
    "\n",
    "**Specifying GPU.** To store the tensor on the GPU, specify `device='cuda'` when you make it, for example `identity_matrix = torch.eye(5, device='cuda')`.  (Instead `device='cpu'` indicates the default CPU storage).\n",
    "\n",
    "Even on a multi-GPU machine it is fine to pretend there is only one GPU.  Setting the environment variable `CUDA_VISIBLE_DEVICES=3` before you start the program will set up the process to see GPU\\#3 as the only visible GPU when it runs.\n",
    "\n",
    "As an aside, in principle you could instead target one of many GPUs with `device='cuda:3'`, but if you want to use multiple GPUs for the same computation your best bet is to a use a multiprocess utility class that manages data distribution between forked processes automatically, while each python process touches only one GPU.  When this becomes an issue, read the [DistributedDataParalllel docs](https://pytorch.org/docs/stable/distributed.html).\n",
    "\n",
    "**Copying a tensor to a different device or type.** You cannot directly combine tensors that are on different devices (e.g., GPU vs CPU or different GPUs); this is similar to how most different-data-type combinations are also prohibited. In both cases you will need to convert types and move devices explicitly to make tensors compatible before combining them.  The `x.to(y.device)` or `x.to(y.dtype)` function can be used to do the conversion.\n",
    "\n",
    "There are also commonly-used convenience synonyms `x.cpu()`, `x.cuda()`, `x.float()`, `x.long()`, etc. for making a copy of `x` with the specified device or type.  There is a bit of cost, so move data only when needed.\n",
    "\n",
    "**GPU rounding is nondeterministic.** Computationally the GPU is **not** perfectly equivalent to the CPU.  To speed parallelization, the GPU does not do associative operations such as summations in a deterministic sequential order.  Since changing the order of summations can alter rounding behavior in fixed-precision arithmetic, GPU rounding can be  different from CPU results an even nondeterministic.  When the numerical algorithm is well-behaved, the difference should be small enough that you do not care, but you should know it is different. You can see this gap in the code example below.\n",
    "\n",
    "**float is fastest.** All commodity GPU hardware is fast at single-precision 32-bit floating-point math, about 20x CPU speed.  Be aware that only expensive cards are fast at 64-bit double-precision math. If you change `torch.float` in the below example to `torch.double` on an Nvidia Titan or consumer card without hardware double-precision support, you will slow down to just-slightly-faster-than-CPU speeds.  Similarly 16-bit `torch.half` or `torch.bfloat16` or other cool options will only be faster on newer hardware, and with these data types you need to take care that the reduced precision is not damaging your results.\n",
    "\n",
    "So `float` is the default and usually the best.\n",
    "\n",
    "Also note that some operations (like linear algebra) are floating-point only and cannot be done on integers.\n",
    "\n",
    "An example of some CPU versus GPU speed comparisons is below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "time using the CPU alone: 1.39 seconds\n",
      "time using GPU, moving data from CPU: 0.135 seconds\n",
      "time using GPU on pinned CPU memory: 0.0728 seconds\n",
      "time using the GPU alone: 0.0174 seconds\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAkEAAAEpCAYAAACUS/YHAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAABcSAAAXEgFnn9JSAAAoyElEQVR4nO3debhkVXno/+8r80xHwKgMDY1iaEVFHECGVoZERREF54uAmvycQNBw1SDXJJqrRlH4KcFoAGeZBUQJMjSjCZExMqexGWQQaMYGmsH3/rFW0UVZZ16nT5+u7+d56tlda9hrV+2qOm/vvYbITCRJkgbNs6b6ACRJkqaCQZAkSRpIBkGSJGkgGQRJkqSBZBAkSZIGkkGQJEkaSAZBkiRpIBkESZKkgWQQJEmSBpJBkCRJGkgGQZIkaSAZBEmSpIFkECRJkgaSQZAkSRpI0z4Iiog/j4ivR8QNEfFoRCyIiEsj4itDlN8rIi6JiIdr2V9ExDYjtLFNLbeg1rskIt4/Oa9IkiQtCZGZU30M4xYRWwO/ANYGrgF+C6wBbA6sn5nL95Q/FDgAeBQ4E1gZ2BEIYM/MPLlPG7sDx1MCxvOBe2qdtYGvZ+aBk/DSJEnSJJu2QVBEPA+4GlgJeG9vABMRr8rMS7qevx44G7gX2Dozb6zpWwNzKYHRxpl5X1edGcDvgLWAt2fmSTX9OcCFwKbA6zPz3Ml6nZIkaXJM59thX6JcjTmo3xWc7gCo+mTdfqETANVyvwaOpAQ6+/bU+WBNP6UTANU6dwEH1adeCZIkaRqalleC6hWaO4DHgD/PzMdGKL8ycD/lqtEGmXlbT/52lFtd52XmnK7084Dtgf+VmT/sqbMi8EB9OmOkY5AkSUuX5UcuslR6LSWgOQt4IiL2ALYFVgCuA46rV2s6XlTL390bAFWX1e0WPelb9OQ/LTMfj4jfAlsBmwFXjvO1EBF3AqsCt453H5IkDagNgEcy88/HWnG6BkGz6/Yu4AJg6578/xsR+2Tm8fX5hnXbLwAiMxdGxP3AjIhYIzMfiog1KbfbhqxX07eq+x93EASsutJKK60xa9aszSewD0mSBs68efNYtGjRuOpO1yBoRt3uBSwCPgCcCqwOfJzST+eHEXF9Zl5V0wEeGWafCylBz+rAQ111hqu3sG5XHyL/GSLi6iGyVpw1axZXXz1UtiRJ6mf27Nlcc80147qTMl07Ri9Xt8sDB2bmUZl5T2bOz8xPAicAK7K483LU7XAdoGKE56OpI0mSponpeiXoobr9I/C9PvlHAXsAc3rKrzbMPlet24d76nTyHhxFnWFl5ux+6fUKkbfCJElagqbrlaD5dXtnZva7EdjJX69ub6nb9fvtLCJWo9wKuz8zHwLIzAdZPPqrb72u9FuGyJckSUup6RoEXV63MyKi3y2pZ9dt5wrN9ZS+Q+tGRL+AZsu6vaon/cqe/KdFxArAi+t+rx/lcUuSpKXEtAyCMvO/KTM5rwK8uk+ROXV7WS3/KHBOTdujT/lO2s970k8fps6ulGU3znaOIEmSpp9pGQRVX67bwyNinU5iRLyCxbNDH9lV/tC6PTgiXtBVfmvgbyh9fv6tp43v1vTdIuJtXXXWAzoLtB6KJEmadqZrx2iA71AWMt0TuD4iLqYMVd+GMjLsO5l5QqdwZp4VEYcB+wNXRMSvarmdKcHgezNzQXcDmbkgIvYFjgNOqDNI3wPsROlDdHhmnj25L1OSJE2GaRsEZeYfI+JdlMVPPwi8njIE/jfAkZn5gz51PhERVwAfowQ/T1AWVf1CZl44RDsnRsT2wMHAayiB07XAtzLz6NavS5IkLRnTNgiCEggBR9THaOscAxwzxnYuAt4wljqSJGnpNq2DIEmSWpr56dNHLqQJmf+lN031ITxtOneMliRJGjeDIEmSNJAMgiRJ0kAyCJIkSQPJIEiSJA0kgyBJkjSQDIIkSdJAMgiSJEkDySBIkiQNJIMgSZI0kAyCJEnSQDIIkiRJA6lZEBQRsyPikIh4+TBlXl7L/EWrdiVJksaj5ZWg/YG/A+4cpsydwMHAxxu2K0mSNGYtg6AdgMsz846hCtS8y4DXNWxXkiRpzFoGQesDvxtFufm1rCRJ0pRpGQQ9DqwxinKrA9mwXUmSpDFrGQRdDWwXEWsPVSAiZgDbAdc2bFeSJGnMWgZBP6ZcCTo+Ip7bm1nTjqVcCfpRw3YlSZLGbPmG+/pX4F3AjsCNEfELYB7l1temwBuBVYGLgX9p2K4kSdKYNQuCMvPJiPgr4HDg/cAePUWeAo4G9s/MJ1q1K0mSNB4trwSRmY8AH4yIgynD4DeoWbcCc4cbPi9JkrQkNQ2COjLzTuAnk7FvSZKkFiYlCAKIiBcA6wD3ZuYNk9WOJEnSeDRdQDUiVomIL0fEvcB1wIXAp7vy94mIyyLiZS3blSRJGquWC6iuBpwHfApYBJwORE+x84GXAe9s1a4kSdJ4tLwS9L+BrYDvABtn5lt6C2TmPMoVop0atitJkjRmLYOgd1LWBftoZi4aptzNuHaYJEmaYi2DoA2BSzPzqRHKPQjMaNiuJEnSmLUMghZSRoONZGPg3obtSpIkjVnLIOhS4FURscFQBSJiNvBy4NcN25UkSRqzlkHQN4FVgJMiYtPezIjYCPh+bfObDduVJEkas2ZBUGaeBnwdeAVwfUT8lrJ46i4R8RvgRspVoK9k5txW7UqSJI1H08kSM/OTlJXk/xvYnDJP0POALSkryv+vzPxMyzYlSZLGo/myGZl5HHBcRKwLbAQsB9yWmb9v3ZYkSdJ4TdraYZl5N3D3ZO1fkiRpIiYtCOqIiOWBDwAvAW4Bvp2ZD0x2u5IkScNpuXbYIRHxVETs0JUWwDnAEcBHgP8L/FdErNmqXUmSpPFo2TF6Z+D3mXleV9rbgG0pHaX/BjgZ2BT4aMN2JUmSxqxlELQJcG1P2h6UYfLvyszvAHtSbont2bBdSZKkMWsZBD2bP+0IvR1wQ2ZeB5CZCfyGMmpMkiRpyrQMgu4G1u08iYhNKHMEnddT7nFgxYbtSpIkjVnLIOgaYLuutcM+RLkV9ouecjOBOxq2K0mSNGYtg6BDgZWBqyLiMuB/A78DzugUiIi1KLNHX9mwXUmSpDFruXbYv1OGwT8AbAZcCOyemY93FduLcivs7FbtSpIkjUfTyRIz80jgyGGKfJeykvzDLduVJEkaq0mfMbpbZj4KPLok25QkSeqn6SrykiRJ04VBkCRJGkgGQZIkaSAZBEmSpIFkECRJkgZSsyAoItaMiDVa7U+SJGkytbwSdD9wZsP9SZIkTZqWQdADwE0N9ydJkjRpWgZBlwOzGu5PkiRp0rQMgr4MvDIi9mi4T0mSpEnRMgh6lLI22LERcUpEfDAidomI7fs9GrZLRPxZRPwhIjIirhuh7F4RcUlEPBwRCyLiFxGxzQh1tqnlFtR6l0TE+1u+BkmStGS1XDtsLpBAAG8Gdh2h/HIN2z4UWGekQhFxKHAAJWA7E1gZ2BnYJSL2zMyT+9TZHTieEjCeD9wD7AgcExEvzcwDm70KSZK0xLQMgr5PCYKWqIjYEXg/8K/AXw9T7vWUAOheYOvMvLGmb00J4I6OiLmZeV9XnRnA0ZSA7e2ZeVJNfw5wIXBARJyWmedOxmuTJEmTp1kQlJl7t9rXaEXEKsCRwDXAVxkmCAI+Wbdf6ARAAJn564g4EtgP2Bf4WledDwJrAad0AqBa566IOAg4CTgQMAiSJGmame4zRv8fyoi0DwNPDFUoIlam3MICOKFPkU7am3vSd+3J73Y68BiwU92/JEmaRiYlCKodlXeOiHeP1Ol4Am1sQbm6c3Rmnj9C8RcBKwF3Z+ZtffIvq9stetK36Ml/WmY+DvyW0q9os9EetyRJWjo0DYIi4jkRcSxwF3AG8EPKLaVO/kfqCKvtJtjOs4DvUGapPmgUVTas234BEJm5sO5rRmfpj4hYE1h7uHpd6RsOkS9JkpZSzfoERcQ6wMXAxpSJEy8CPtZT7GfAYcAewAUTaO7jwKuAfTLz3lGUX71uHxmmzEJK0LM68FBXneHqLezZ/7Ai4uohspxkUpKkJazllaDPUQKgQzLzFZm5X2+BzLwduBYY9zxBEbEB8AXgvMw8ZrTVOocwijJDPR9NHUmSNE20HCL/FuDazPzCCOVuBl4zgXaOAFakdIYerYfqdrVhyqxatw/31OnkPTiKOsPKzNn90usVos1Hsw9JktRGyyDoucApoyj3GLDGBNrZldJ/518innEhpjNCa8OImNspm5kPA7fU5+v322FErEa5FXZ/Zj4EkJkPRsQDlCHy61OG4ffq7O+WPnmSJGkp1jIIegB4/ijKvQC4c4JtrQ3sMETeKl15ndd3PbAIWDci1u8zQmzLur2qJ/1Kyq27LekJgiJiBeDFdb/Xj/H4JUnSFGvZJ+hi4FUR0feWD0BEvJYy7HykIe1Dyszo96D0RwK4viv9/lrnUeCcmt9vgddO2s970k8fps6ulKtPZ2fmY+N9PZIkaWq0DIK+Rlle4tSI2LEOY39aRGwL/AB4Evh6w3ZH69C6PTgiXtB1XFsDf0Pp8/NvPXW+W9N3i4i3ddVZD/hKz34lSdI00iwIyswLKWtzbURZnHQBZTTW2yLibuA8ynw6n8jMy1u1O4bjO4syPP/ZwBUR8bOI+AXlqtQKwL6ZuaCnzgLKUhp/BE6IiHMj4njK7a9NgcMz8+wl+TokSVIbTSdLzMzDgW2B0+q+A1iTMo/OmcDrMvOIlm2O8fg+AexDGaa/M7ANcDawQ2aeOESdEyn9gv4deBnwRmAeJWjaf/KPWpIkTYaWHaMByMz/AN4aZejWsym3yO7JzKdat9XT7nxGMW9PnVvomDHu+yLgDeM5LkmStHRqHgR1ZGYC90zW/iVJkiZiUoKgiHg15bbY82rS7cBF9SqRJEnSlGsaBEXES4CjWDzvzjOWq4iIKyjrffXOxyNJkrREtVxAdTPKCLC1gVuBE4H5lEBoQ+DtwMuB8yJi68y8rlXbkiRJY9XyStA/UQKgL1EWUX2yOzMiDgL+AfgM8EVKUCRJkjQlWg6Rfx1wdWZ+tjcAAsjMpzLz74Cra1lJkqQp0zIIWoE/XXurn6tqWUmSpCnTMgi6Epg1inKzallJkqQp0zII+iLwyojYd6gCEbEP8EpK/yFJkqQp07Jj9ELgX4DvRMTewLHAzTVvI+CdwGtrmYcjYvvuypk57pXlJUmSxqplEDSXMh9QUCZKfG1PfmfOoA/XR6/lGh6LJEnSsFoGQd+nToooSZK0tGsWBGXm3q32JUmSNNladoyWJEmaNgyCJEnSQDIIkiRJA8kgSJIkDSSDIEmSNJAMgiRJ0kAyCJIkSQPJIEiSJA2kZkFQRDwnIraPiOf0pG8cET+JiN9GxOkR8apWbUqSJI1XyytBnwbOBdbuJETE6sCFwDuAzYE3AGdHxCYN25UkSRqzlkHQHODazLy+K21v4LnAT4DNgAOA1YBPNWxXkiRpzFoGQc8HbupJ2xV4Etg/M2/MzMOAK4DXNWxXkiRpzFoGQWsAD3WeREQArwYuzcx7u8pdD6zfsF1JkqQxaxkE/R7YuOv5VsBawNyecssDjzdsV5IkacxaBkG/Bl4VEbtFxJrAwUACp/WU+wtKwCRJkjRlWgZBXwQWAScB9wFvBuZm5sWdAhExkzJK7D8btitJkjRmy7faUWZeFxHbAvsD6wKXAv/cU+wvgSuBn7VqV5IkaTyaBUEAmXk5ZVj8UPnfBr7dsk1JkqTxcNkMSZI0kJpeCeqIiA0pkySuNFSZzDx/MtqWJEkajaZBUETsC3wO2HAUxZdr2bYkSdJYNAuCImIf4Lv16X8DNwAPt9q/JElSSy2vBB1IWSLj7ZnZOzeQJEnSUqVlx+gXAOcbAEmSpOmgZRC0AG9/SZKkaaJlEHQKZdmMVRruU5IkaVK0DII+CzwIHBMRazfcryRJUnMtO0Z/DbgG2APYJSJ+A9xGWUS1V2bmBxq2LUmSNCYtg6C9u/69FrDjMGUTMAiSJElTpmUQ9LqG+5IkSZpULVeRP6/VviRJkiabC6hKkqSB1HwB1YhYAdgd2A54HqX/zx3ABcDJmflE6zYlSZLGqvUCqq8FfgysD0RP9keAWyPiPZl5cct2JUmSxqrlAqovBH4JrA5cCvwQmF+zNwLeB2wF/DIitsrMG1u1LUmSNFYtrwT9HSUAOiAzD+uTf3hE7Ad8o5bdu2HbkiRJY9KyY/SOwOVDBEAAZObhwOXATg3blSRJGrOWQdC6wHWjKHcdsE7DdiVJksasZRB0L/DCUZR7IWXFeUmSpCnTMgg6F9gyIj40VIGa9wrgnIbtSpIkjVnLjtFfAN4KHBkR76EMlZ9PmSdoY+C9lLmDHgG+2LBdSZKkMWu5bMa1EfEW4EfADsD2PUUCuAt4b2Ze26pdSZKk8Wg6WWJmnh0RmwDvYPGM0QC3U2aMPi4zH2nZpiRJ0ng0XzajBjnH1IckSdJSyQVUJUnSQBp3EBQRG9bHcj3PR/WYyEFHxKoR8daI+LeIuCoiHoyIhRFxZUQcEhGrD1N3r4i4JCIejogFEfGLiNhmhPa2qeUW1HqXRMT7J/IaJEnS1JrI7bD5wB+BzYEbWDwSbDRygm2/B/hO/ffVwBnAmsA2wN8D746IHTLzD92VIuJQ4ADgUeBMYGVgZ2CXiNgzM0/ubSgidgeOpwSM5wP3UGbHPiYiXpqZB07gdUiSpCkykUDkfEow80jP8yXhceBfgK93L8QaEc8FTgdeTlmj7D1dea+nBED3Alt36kXE1sBc4OiImJuZ93XVmQEcDSwHvD0zT6rpzwEuBA6IiNMy89zJe6mSJGkyjDsIysw5wz2fTJn5feD7fdLviIiPAhcDb4uIFTPz8Zr9ybr9QnfglJm/jogjgf2AfYGvde3yg8BawCmdAKjWuSsiDgJOAg6kTBQpSZKmkWWxY/SVdbsS8GyAiFiZcgsL4IQ+dTppb+5J33WYOqcDjwE71f1LkqRppFkQFBHn1KsjI5X7VERM5rIZm9TtEyxeo+xFlKDo7sy8rU+dy+p2i570LXryn1avMP2W0q9os4kcsCRJWvJaXgmaQwk2RrIZZUbpybJ/3Z6RmYvqvzuj0foFQGTmQuB+YEZErAEQEWsCaw9Xryt9QqPdJEnSktd8ssRRWBl4cjJ2HBFvBD5AuQr0ua6szpD54WarXkgJelYHHuqqM1y9hT37H+n4rh4ia9Zo6kuSpHaWaJ+genVlG+COSdj3XwA/pKxR9reZeWV3dt0ON3otRng+mjqSJGmamNCVoIi4qSdpj4iYM0xbz6nbb06k3T7HsT5lrqAZwKGZeVhPkYfqdrVhdrNq3T7cU6eT9+Ao6gwrM2f3S69XiDYfzT4kSVIbE70dNrPr30m5LTTUraEnKAupngp8ZoLtPi0i1gF+RemXczTwqT7Fbqnb9YfYx2qUW2H3Z+ZDAJn5YEQ8QBkivz5wTZ+qnf3d0idPkiQtxSZ0Oywzn9V5UG4NHdOd1vNYKTM3zsz9W60kXzsx/5LSIfsk4EOZ2e+W1/XAImDdetWo15Z1e1VP+pU9+d1trwC8uO73+rEfvSRJmkot+wTtA/xbw/0NKyJWAk4BtgL+HXh3Zj7Vr2xmPgp0huXv0adIJ+3nPemnD1NnV0on77Mz87ExHLokSVoKNAuCMvN7mXlRq/0Npy7a+hPgdcAFwNu6ZoYeyqF1e3BEvKBrX1sDf0Pp89MbxH23pu8WEW/rqrMe8JWe/UqSpGlkUobI19tUs4A1GGIEVWaeP4EmPgbsXv99D3BERN9mPpWZ99T2zoqIwyjzCF0REb8CVqQsoPos4L2ZuaC7cmYuiIh9geOAEyLivNreTpQ+RIdn5tkTeB2SJGmKNA2CIuLFlIVL5zDy8PHlJtDUjK5/7z5kKfg8JWgBIDM/ERFXUIKonSmdtc+mrCd2Yb8dZOaJEbE9cDDwGkrgdC3wrcw8egKvQZIkTaFmQVC9xXQhsCZwEfBcYGPgp5SlLLas7Z1KmZ153DLz85QAZzx1jwGOGWOdi4A3jKc9SZK0dGrZMfpgyu2vfTJzO0pfHTLzvZm5NTCbEiRtTll5XZIkacq0DIJeD1ybmd/rl5mZ/wPsBqwL/GPDdiVJksasZRC0Hs+cUPAJgIhYuZOQmfcDcynDyyVJkqZMyyBoAWXenO7nABv1Kbtew3YlSZLGrGUQ9DtKR+iOKygjxN7VSahLXMzBZSYkSdIUaxkEnQlsHhGdQOg0yvD0QyLi2Ij4GvBflLW4jmvYriRJ0pi1nCfoB8BKlI7Pv8vMhRHxLkrAs2dXuV8BX2zYriRJ0pg1C4Iycx49q8Nn5jkRsRGwHWWCwxsy89JWbUqSJI1Xy8kS3wI8kZm/7E7PzIXAGa3akSRJaqFln6CTgf0a7k+SJGnStAyC7gbua7g/SZKkSdMyCJoLvCqGWM5dkiRpadIyCPocsA7w9e5ZoiVJkpZGLYfIvxv4BfBx4F0RcRZlUsTH+pTNzHT9MEmSNGVaBkGfB5IyS/R6wHuGKZu4iKqkZdDMT58+1YewzJv/pTdN9SFoGdEyCNqn4b4kSZImVcvJEr/Xal+SJEmTrWXHaEmSpGmj5e0wACJieWBX4JWU0WL/mZlH1bzn1bRrMvPJ1m1LkiSNVtMgKCJ2oCyk+nxKB+kEVgCOqkV2BI4B3gGc2LJtSZKksWh2OywiXkIZIr8ecBhl5fjeiRNPBB4B3t6qXUmSpPFoeSXoEGAlYJfMPAegd/LozHwkIq4FXt6wXUmSpDFr2TF6B+A/OgHQMG4BntewXUmSpDFrGQStCfx+FOVWApZr2K4kSdKYtQyC7gD+YhTlXgzc3LBdSZKkMWsZBJ0JzI6I3YcqEBF7AxsBzisvSZKmVMsg6J+Ah4GfRMQ/RsRWNX3ViHhxRBwMHAHcCxzasF1JkqQxaxYEZebNwJuA+4C/A/6TMk/QnsCVwD8ADwG7ZeadrdqVJEkaj6aTJWbmhRHxQuADwE7ATEon6NuAs4BvZ+b9LduUJEkaj+bLZmTmQ8A36kOSJGmp1HLG6EMi4i2jKPfmiDikVbuSJEnj0bJj9OeBt46i3FuA/9OwXUmSpDFrGQSN1nLAH6egXUmSpKdNRRA0mzKCTJIkacpMqGN0RBzVk7Rtn7TutjYDtgJ+NpF2JUmSJmqio8P27vp3ApvWx3CuAv52gu1KkiRNyESDoNfVbQDnAGcAXx6i7OPA7XVSRUmSpCk1oSAoM8/r/Dsivgdc0J0mSZK0tGo2WWJm7tNqX9Kgm/lp1xiebPO/9KapPgRJU2wqRodJkiRNuXFfCYqImyidoXfKzN/V56OVmTlrvG1LkiRN1ERuh82s2xV6nkuSJC31xh0EZeazhnsuSZK0NDNwkSRJA8kgSJIkDSSDIEmSNJAMgiRJ0kAyCJIkSQPJIEiSJA0kgyBJkjSQDIIkSdJAMgiSJEkDySBIkiQNJIMgSZI0kAyCJEnSQDIIkiRJA8kgSJIkDSSDIEmSNJAMgkYhIlaOiL+PiBsi4rGIuD0ijoqI9af62CRJ0vgYBI0gIlYGzgYOAVYHTgFuBfYBLouIWVN4eJIkaZwMgkb2WWAb4NfACzPznZn5auCTwLrAUVN5cJIkaXwMgoYRESsAH69PP5qZD3fyMvNQ4Cpg+4h4xVQcnyRJGj+DoOFtC6wNzMvMy/vkn1C3b15iRyRJkppYfqoPYCn30rq9bIj8y3rKLVVmfvr0qT6EZd78L71pqg9BkjROBkHD27Bubxsi/7aecsOKiKuHyHrRvHnzmD179liObUS33/XwyIU0IbNPW31S9uu5m3yeu+lrss4deP6WhNbnb968eQAbjKeuQdDwOmfqkSHyF/aUG68/Llq0aOE111xz6wT3M511RtnNm9KjGKNr7p3qI1hqTLvz57l7mudu+pp25w4m5fxtwNB/p4dlEDS8qNscIX9UMrPtpZ5lSOcqme/R9OT5m748d9OX527i7Bg9vIfqdrUh8letW6+fSpI0zRgEDe+Wuh1qZuj1e8pJkqRpwiBoeFfW7ZZD5HfSr1oCxyJJkhoyCBreRcADwKyIeHmf/D3q9udL7pAkSVILBkHDyMzHgW/Wp9+MiKf7BkXEgcAWwIWZ+V9TcXySJGn8InOogU+CpxdQnQu8GrgDuADYqD6/F3hNZv7PlB2gJEkaF4OgUYiIVYDPAO+hzEdwH3AG8LnMHOS5fSRJmrYMgiRJ0kCyT5AkSRpIBkGSJGkgGQRJkqSBZBAkSZIGkkGQJEkaSAZBkoYVEXMjIiNi5lQfS7d6TPOn+jiWtKX1fIxXRHy+vp69p/pYNHgMgiRJ0kBafqoPQNJSby9gVeD3U30gAjwfUjMGQZKGlZm3TPUxaDHPh9SOt8PUVERsGBHfjIgbI+KxiLg3Ii6JiM/W5Uee0achIt4XEZdGxCMR8YeI+F5EPL/Pfo+pdeYM0e5A9g/pp76vWd/n1SLi0Ii4NSIejYjLIuLNXWX3rOdnYUTcFRGHd85TV5k/6YMSEV+pacf2aX+9iLgzIp6MiK178l4SET+KiN9HxKKIuD0ijh6qf0s9/i9HxC3183RdRBwYETHR92lp0HOu1oyIw+q5eiwiro2IAyLiWT11+vYJ6nwHImK5iDgoIm6o7/Gt9T1cqU/78yMi678/GBFX1c/JnRHx7YhYe4jjXjEi9o+I/4qIh+rn55KI+MBQ5yYidqjH/nD9XTg5Il403vduWRAR76jv4aP1+3d0RDyn3+9d1/ldMSL+PiLm1c/JTRHxD1HWuezd/9Pnt0/enLrPYybtBU4HmenDR5MHsD1wP5DAPOBY4OfATTVtZi03tz7/JvBH4DzgJ8DvavqtwPo9+z6m5s0Zou0E5k/1e7A0PICZ9f24GPgP4G7gNOBc4CngSWAn4ADgiVruZOCeWu9HPfvrnK+ZXWkrApfV9L16yp9W0z/fk/52YFHN+w1wfNc+7gFm95RfCbio5t9dy58BPA58a1k4513n6tf1PbkPOLG+h4/UvKNHOh81PYH5wE+Bh4Fz6n4638kf9ml/fs37Sj03F9bPwl01/Xzq8kpddVar6Z3z8kvgdGBBTTuyTzu71c9d1nP6E8pvxAPAD2v63lN9Ppbwuf9Efd1PAmfV83Yb5XfwFHp+7+rzm4FT62fjtPpZ6Zzfs4Dl+p3fIdqfU+sdM9XvxZSeh6k+AB/LxgOYAfyhfqk+0eeHc3tgrfrvzo/4E8Abu8qs0PWDeFJP/WN6fxR68qf9H8SG56LzhzUpgc+Mrry9a/qNwL3Adl15z+v647dJV3rnfM3saedF9cf4ARYHuB9m8R/15bvKbgwsrD/Y2/fsZ69a55Ke9M/U9P/sfHZq+pa1zWl/znvO1ZXAOl15syj9fhJ4yyjOR2c/1/DMgHVjFgcos3rqzK/ptwMv60pfp35GEnh9T50javr3gdW70telBN0JvKkrfQ1KsJTAu7vSl+/6Xg9UEARsQgk6H+35Dq5M+Y9j5z2Z0+f83trz/VwX+O+at1+/8zvEMczBIMggyEebB3BQ/UKdNoqynR/xH/XJezblf7FPAc/vSu/8WM4ZYp/T/g9iw3PR+cP6JLBpT96zWBys/n2fuof2/kEa6o9uzftIzbsAmE0JdB7q88f2G7XcXw9xzCfX/C270m6padv0Kf9Py8I555lB0M598v+/mvfvI52Prv3s2Gc/h/cLNFgcBH2gT50D6bmiB6xHuRJ3E7BSnzovrXVO7Urbt6ad2af8jPp5GbQg6AsMfdVsVv39GyoI+lCfOn9V867vd36HOIY5GATZJ0jN7FS33x5DnZ/2JmTmvcCvKH+st2lwXINsfmb+T3dCZv6Rckkdyvvca17dPnc0DWTmEZRbIdtSbqutCuyfmfN6iu5ct6cMsasL6/aVUPqWARsAv8/Mi/uU/8lojm8aWZCZ/c7Hj+t2m1H2g3qCEiT1uqFuhzqvZ46yzg6UK7ZnZOai3gqZeSUlqHllV/K2dXtcn/L3DdH2sq7z23Z8b0b97lw+TN1+v5tnUG6lvjAi1m1yhAPCIEitbFC3vX/8hnPzEOnz6/Z54z4awdBDqBcOk9/J+5NOtMP4APAYsCblSuBRfcrMrNs7a2fMZzyAr9b8deq2c+6HGgm1rI2Q6vtdyMwHKbcQV6e8vyO5IzOf6pP+cN0OdV5vG2WdmXX74X7nsZ7LNVh8HmHwzuVodN6TW4fIH+o9uS8zHxoir/MZ8ndzDBwir9aywT7GNPKnd/SMnjbSuWhxrgDeQunLAPCiiFgtMxf2lFmOxf1IhnN13XY+A0MdY6tjnw7G8n0Y1/uS9f7IKCxXt5cDV42yzkjncpAN9Z6MZ/TjWOv4u4lBkNq5ldJRdlPgulHW2Yj+P6Qb1u3tXWmP1+3qfcpv0CdNS0BEvAD4OuUK0q+At9bnf91T9DZKX4f96tWNkXTO/UZD5A+VPl1t2C8xItYE1qK8v6N53yZb54rR3Mw8cJR1RjqXfV/7Mu4OYDPKa7+xT/5Qv2kzImKNIa4Gdd7HO7rSHgeIiNUz8+Ge8v5uYiSods6q294/fsN5Z29CRPwZsAuLRxh1dL7YL+yzn13G0KYaiYjlgR9Rhkx/Angf5Qf9QxHx1p7inc9Hb3pfmXkz5Q/u83vnGqreNfYjXqo9OyJ26pP+7rq9eAxXaybTuZROu7tGxHIjFa46/b327M2o8xAN4ve3089tj96MiNgEePkwdfv9bv4lpZP5jZn5h64sfzdHYBCkVr5LmevlzRHxsd5OnBGxXUSs1VPnHfXL2ymzPOUqwmqU0SXd/RTOq9sPR8Szu+psCfxjw9eh0fs8pQPsKZn53XoL7H2UUWnfiYg/7yr7Ncpw4K9H12SNHRHxZxHxkXjmRI2dTvZfq1dEOmVfBny06StZOvxzz2d7Y+Bz9ekRU3NIz5SZv6eM1HwB8IOIWKe3TERsExFv7Eo6njJEf5eIeEdXueUon4t+V3eXdUdTOrHvHRFPDwCpEx5+g+H/Nh8Sz5y4dB3KPE/wp5+Tzu/mZ7qD1oh4H8vefyTGxSBITWTmAuAdlJEh/z9wY0QcGxGnRcRNlMnVZvRU+1fgl3UW2R9TRqPsRbl8vl9P2XMpX+hNgWsi4qSIuIAyL8kPJut1qb+I2Bb4NGVeoQ910jPzEkpQug5wdCcYzswbKQHSKsCpUWZ+PjkifhYRl1P+x/otntkJ958pcwRtDcyLiOMi4pc17ccsW/6DMnHojRFxQkScCvwWeD5lksOfTeXB9diP8n18N3BTRJwfET+t3+PbKJMhPn2Vod7+/GvK6zs2Ii6s3/frKVdCfrTEX8EUq6M2P0vpS3d+RPwqIn5KuZL6UspEiLC4G0DHLZT5pK6OiFMj4sRaZwvKOflmT/lvUeZo2oPyu3l8RFxBCWQPa/26piODIDWTmecCL6MEN8tTbn28hjIvzWeAO3uqfBXYh9LnYXfK6JcfAK/OnvWR6q2A3YAjKbfK3kgJqvbLzL+dlBekvupVmR9QOsnum5l39xT5IuVW5l8BH+skZuZJlB/4b1OGWb+BMlfJSpQ/hLtSJkHslF9EmXrhq5SJ5XajTDJ3cPd+lxGLgNdThv5vDfwlpZ/dpygTXC41MvMRSpDzQcqM3y+mfH9nUUaHHsTi0X6dOidSpkm4gHKr5w2USR23Bp4xjcOgyMyvUgLJK4HtKOf/XMpvZueK6L291SgBzTeAl7D4O/NFygSVT/a0cRdlotqfU6Y6eEMtvzNl5umBF0vHbWYNkoiYS5lvZOPMnD+1RyNNnXpb43fAeZk5Z2qPRkuDiFiNMk3IKpSZ0p+q6QncnJkzp+7olj1eCZIkaQmLiE16+0lGxOqUq93rAMcOMeeTGnKIvCRJS947gM9HxKWUkZAzKLcK16FcCfrs1B3a4DAIkiRpyTub0ofyNZTgJygdn78HfLlPXztNAvsESZKkgWSfIEmSNJAMgiRJ0kAyCJIkSQPJIEiSJA0kgyBJkjSQDIIkSdJAMgiSJEkDySBIkiQNJIMgSZI0kAyCJEnSQDIIkiRJA8kgSJIkDSSDIEmSNJAMgiRJ0kD6f2fzpPF1P8mgAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 600x300 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Your GPU is 80.1x faster than CPU but only 10.3x if data is repeatedly copied from the CPU\n",
      "When copying from pinned memory, speedup is 19.1x\n",
      "Numerical differences between GPU and CPU: 0.0002938236575573683\n"
     ]
    }
   ],
   "source": [
    "import torch, time\n",
    "from matplotlib import pyplot as plt\n",
    "\n",
    "# Here is a demonstration of moving data between GPU and CPU.\n",
    "# We multiply a batch of vectors through a big linear opeation 10 times\n",
    "r = torch.randn(1024, 1024, dtype=torch.float)\n",
    "x = torch.randn(32768, 1024, dtype=r.dtype)\n",
    "iterations = 10\n",
    "\n",
    "def time_iterated_mm(x, matrix):\n",
    "    start = time.time()\n",
    "    result = 0\n",
    "    for i in range(iterations):\n",
    "        result += torch.mm(matrix, x.to(matrix.device).t())\n",
    "    torch.cuda.synchronize()\n",
    "    elapsed = time.time() - start\n",
    "    return elapsed, result.cpu()\n",
    "\n",
    "cpu_time, cpu_result = time_iterated_mm(x.cpu(), r.cpu())\n",
    "print(f'time using the CPU alone: {cpu_time:.3g} seconds')\n",
    "\n",
    "mixed_time, mixed_result = time_iterated_mm(x.cpu(), r.cuda())\n",
    "print(f'time using GPU, moving data from CPU: {mixed_time:.3g} seconds')\n",
    "\n",
    "pinned_time, pinned_result = time_iterated_mm(x.cpu().pin_memory(), r.cuda())\n",
    "print(f'time using GPU on pinned CPU memory: {pinned_time:.3g} seconds')\n",
    "\n",
    "gpu_time, gpu_result = time_iterated_mm(x.cuda(), r.cuda())\n",
    "print(f'time using the GPU alone: {gpu_time:.3g} seconds')\n",
    "\n",
    "plt.figure(figsize=(4,2), dpi=150)\n",
    "plt.ylabel('iterations per sec')\n",
    "plt.bar(['cpu', 'mixed', 'pinned', 'gpu'],\n",
    "        [iterations/cpu_time,\n",
    "         iterations/mixed_time,\n",
    "         iterations/pinned_time,\n",
    "         iterations/gpu_time])\n",
    "plt.show()\n",
    "\n",
    "print(f'Your GPU is {cpu_time / gpu_time:.3g}x faster than CPU'\n",
    "      f' but only {cpu_time / mixed_time:.3g}x if data is repeatedly copied from the CPU')\n",
    "print(f'When copying from pinned memory, speedup is {cpu_time / pinned_time:.3g}x')\n",
    "print(f'Numerical differences between GPU and CPU: {(cpu_result - gpu_result).norm() / cpu_result.norm()}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise\n",
    "\n",
    "Repeat the benchmark using type `torch.double`.   What does that tell you about your GPU hardware?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO: Repeat the benchmark using type torch.double.\n",
    "r = 'TODO'\n",
    "x = 'TODO'\n",
    "\n",
    "# Benchmark and plot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Performance tips\n",
    "----------------\n",
    "\n",
    "**GPU operations are async.** When pytorch operates on GPU tensors, the python code does not wait for computations to complete. Sp GPU calculations get queued up, and they will be done as quickly as possible in the background while your python is free to work on other things like loading the next batch of training data.\n",
    "\n",
    "**Moving data to cpu waits for computations.** You do not need to worry about the GPU asynchrony, because as soon as you actually ask to look at the data, e.g., when you move GPU data to CPU (or print it or save it), pytorch will block and wait for the GPU operations to finish computing what you need before proceeding. The call seen above to `torch.cuda.synchronize()` flushes the GPU queue without requesting the data, but you will not need to do this unless you are doing performance timing.\n",
    "\n",
    "**Pinned memory transfers are async and faster.** Copying data from CPU to GPU can be sped up if the CPU data is put in pinned memory (i.e., at a fixed non-swappable block of RAM).  Therefore when data loaders gather together lots of CPU data that is destined for the GPU, they should be configured to stream their results into pinned memory. See the performance comparison above."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "pytorch Tensor dimension-ordering conventions\n",
    "---------------------------------------------\n",
    "\n",
    "**Multidimensional data convention.** As soon as you have more than one dimension, you need to decide how to order the axes.  To reduce confusion, most data processing follows the same global convention. In particular, much image-related data in pytorch is four dimensional, and the dimensions are ordered like this: `data[batch_index, channel_index, y_position, x_position]`, that is:\n",
    "\n",
    "* Dimension 0 is used to index separate images within a batch.\n",
    "* Dimension 1 indexes channels within an image representation (e.g., 0,1,2 = R,G,B, or more dims for more channels).\n",
    "* Dimension 2 (if present) indexes the row position (y-value, starting from the top)\n",
    "* Dimension 3 (if present) indexes the column position (x-value, starting from the left)\n",
    "\n",
    "There a way to remember this ordering: adjacent entries that vary only in the last dimensions are stored physically closer in RAM; since they are often combined with each other, this could help with locality, whereas the first (batch) dimension usually just groups separate independent data points which are not combined much, so they do not need to be physically close.\n",
    "\n",
    "Stream-oriented data without grid geometry will drop the last dimensions, and 3d grid data will be 5-dimensional, adding a depth z before y.  This same 4d-axis ordering convention is also seen in caffe and tensorflow.\n",
    "\n",
    "Separate tensors can be put together into a single batch tensor using `torch.cat([a, b, c])` or `torch.stack([a, b, c])`.  (The difference: `cat` doesn't add any new dimensions but just concatenates along the existing 0th dimension.  `stack` adds a new 0th dimension for the batch.)\n",
    "\n",
    "**Multidimensional linear operation convention.** When storing matrix weights or convolution weights, linear algebra conventions are followed\n",
    "* Dimension 0 (number of rows) matches the output channel dimension\n",
    "* Dimension 1 (number of columns) matches the input channel dimension\n",
    "* Dimension 2 (if present) is the convolutional kernel y-dimension\n",
    "* Dimension 3 (if present) is the convolutional kernel x-dimension\n",
    "\n",
    "Since this convention assumes channels are arranged in different rows whereas the data convention puts different batch items in different rows, some axis transposition is often needed before applying linear algebra to the data.\n",
    "\n",
    "**Permute and view reshape an array without moving memory.** The `permute` and `view` methods are useful for rearranging, flattening, and unflatteneing axes. `x.permute(1,0,2,3).view(x.shape[1], -1)`.  They just alter the view of the block of numbers in memory without moving any of the numbers around, so they are fast.\n",
    "\n",
    "**Reshaping sometimes needs copying.** Some sequences of axis permutations and flattenings cannot be done without copying the data into the new order in memory; the `x.contiguous()` method copies the data iinto the natural order given by the current view; also `x.reshape()` is similar to `view` but will makea copy if necessary so you do not need to think about it.  See [the Tensor.view method documentation](https://pytorch.org/docs/master/tensors.html#torch.Tensor.view).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise\n",
    "\n",
    "Use `torch.randn` to create a four-dimensional tensor `x` of size (2,3,4,5), which could store two 5x4 RGB images.\n",
    "\n",
    "Then print three things:\n",
    " * print `x`.\n",
    " * Use `x.permute` to switch the horizontal and vertical (last two) dimensions.\n",
    " * Use `x.view` to see each image as a flat vector of 60 numbers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TODO\n",
      "TODO\n",
      "TODO\n"
     ]
    }
   ],
   "source": [
    "# TODO make x of size (2,3,4,5), and print three rearrangements of x\n",
    "x = 'TODO'\n",
    "print(x)\n",
    "print('TODO')\n",
    "print('TODO')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Special topic: einsum notation\n",
    "\n",
    "Matrix multiplication can be generalized to tensors of arbitrary number of dimensions, but keeping tensor dimensions straight can be confusing.  The solution to this is [Einstein notation](https://en.wikipedia.org/wiki/Einstein_notation): assign letter variables to each axis of the input tensors, and then explicitly write down which axes end up in the output tensor.  For example, an outer product might be written as `i, j -> ij`, whereas matrix multiplication could be `ij, jk -> ik`.\n",
    "\n",
    "Einstein notation is a topic of active development and programming language design: [here is a recent paper on the history and future of Einstein APIs.](https://openreview.net/pdf?id=oapKSVM2bcj)\n",
    "\n",
    "\n",
    "In pytorch, Einstein notation is available as `einsum`.  Here is how ordinary matrix multiplication looks as einsum:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[ 3.2591, -0.9139,  3.3531],\n",
      "        [ 4.6914, -1.4011,  5.6399]])\n"
     ]
    }
   ],
   "source": [
    "A = torch.randn(2,5)\n",
    "B = torch.randn(5,3)\n",
    "\n",
    "# Uncomment to see ordinary matrix multiplication\n",
    "# print(torch.mm(A, B))\n",
    "\n",
    "# Ordinary matrix multiplication written as an einsum\n",
    "print(torch.einsum('ij, jk -> ik', A, B))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise\n",
    "\n",
    "Make A in the shape (5, 6, 2) and B in the shape (5, 6, 3); we can think of A as a 5x6 grid of 2-dimensional vectors and B as a 5x6 of 3-dimesnsional vectors.\n",
    "\n",
    "Covariances (un-normalized) of vectors in A and B could be computed by flattening and transposing the tensors into (2,30) and (30,3) matrices and then doing a matrix multiplication of these batches as follows:\n",
    "\n",
    "```\n",
    "print(torch.mm(A.reshape(30, 2).t(), B.reshape(30, 3)))\n",
    "```\n",
    "\n",
    "Instead use einsum to compute the same thing.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TODO\n"
     ]
    }
   ],
   "source": [
    "# TODO: use einsum to compute a covariance statistic over vectors in A and B.\n",
    "A = torch.randn(5,6,2)\n",
    "B = torch.randn(5,6,3)\n",
    "\n",
    "\n",
    "print('TODO')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [On to topic 2: Autograd &rightarrow;](2-Pytorch-Autograd.ipynb)"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
